Analyzing Visitor Evaluations at the Nantucket Historical Association

Executive Summary

Nantucket has a long history from its whaling days to the present. The Nantucket Historical Association (NHA) preserves, interprets, and fosters an appreciation of that history. The NHA offers a wide variety of programs, tours and exhibits to engage visitors and residents. It is best practice for museums to survey their visitors to gain an understanding of the larger picture: are the exhibits interesting and engaging? Are facilities clean and up to visitor expectations? How does the museum portray itself from the beginning to the end of the visitor experience? Bill Tramposch, Director of the NHA, hopes to show his staff that evaluations are not as complex as they appear to be and get everyone involved with future evaluations. In order to do that, the NHA has asked our team to analyze and evaluate their visitor experience evaluation protocols and make recommendations on how to conduct future evaluations in the museum.

Evaluation has become increasingly important in museums. This is due to a desire to learn more about visitor needs and interests in order to design better exhibits, programs, and activities. Evaluations also produce data that can be used to improve the effectiveness of marketing efforts and enhance a museum’s ability to obtain funding through grants. Increasingly, funding organizations require formal evaluation as a stipulation for an award. Today, there are three major sectors in the world of museum evaluation: general museum, exhibit, and educational evaluation. Each of these can help to provide a quality visitor experience and further the goals of the museum. The team focused on educational and exhibit evaluation in this project.

The overarching goal of this project was to help the Nantucket Historical Association assess and enhance the way the museum conducts its visitor evaluation. In order to accomplish this goal, the project team identified several project objectives and various associated tasks. After clarifying the scope of the visitor studies conducted by the NHA, the team coded and analyzed visitor survey data collected during summer 2011. These data, as well as feedback from NHA staff members, were used to recommend improvements in the NHA’s survey instruments and protocols.

The NHA has conducted relatively few visitor evaluations in the past. For this reason, the instruments and protocols they use need to be developed and the staff needs to be iv trained both in the importance of evaluation and the mechanisms of evaluation. Like other museums, the NHA is interested in conducting more visitor evaluations in order to improve exhibits and programs (i.e., the visitor experience) and enhance its marketing and fund-raising efforts. As an institution, the NHA needs to create a sense that evaluation is valued by the organization so that staff will see it as an important part of their jobs, not a distraction from other tasks.

Based on the data and our observations we were able to draw a variety of conclusions:

• From the data it appears that the museum is doing a good job meeting visitor’s needs and expectations. Ninety three percent of the respondents rated their overall experience as 6 or above on an 8-point scale (where 8 equaled ‘extremely satisfied’) to yield an average score of 7.34 overall.
• There is an inevitable tension between the needs of different staff and how they can be met through visitor evaluation (e.g., curators/interpreters vs. marketing). This was seen during interviews with each department about what they wanted to include in the surveys and learn from evaluations. Each visitor survey instrument will need to be designed to balance these different needs, and we have developed some model instruments that try to do this. It may be necessary in the future to develop other instruments with varying emphases on curatorial and programmatic needs versus marketing. In designing these surveys, staff should avoid the temptation to include more and more questions, since the survey instruments will ‘balloon’ and the response rates and quality of information will decline.
• Staff need more training in the development and delivery of surveys and the collection of additional data (e.g., zip codes, etc. at the point of sale). This ranges from simple things such as writing legibly on surveys to more complex things like survey administration and data coding and analysis. Training more staff on appropriate surveying techniques will have a variety of benefits. It will improve the quality and quantity of data that can be collected and reinforce the sense among staff that visitor evaluation is important. Greater involvement of staff in evaluation may also encourage different kinds of interactions between staff and visitors and thus encourage a more reflective assessment of what the museum does and how it does it.
• The NHA’s survey instruments and protocols need improving. The team put a lot of focus into working with staff to improve these evaluation tools. We are confident that the NHA now has a good framework to move forward with and instruments that will obtain useful, higher quality data.
• The more that data entry and analysis can be streamlined, the better able the staff will be able to administer and analyze evaluation surveys. The team learned this from coding the data from the past evaluations, which were in multiple formats and difficult to compare. The team created a coding sheet for them to follow that will work with all future evaluations and keep the data organized and in similar states for easy analysis.
• Technology from the point of sale to hand-held devices offers major opportunities for improved evaluation and ongoing collection of data necessary for marketing. The NHA staff appears eager to weave technology into their surveying and data collection instruments. Implementing improved software at the point of sale would allow the NHA to gather membership, zip code, and email data more easily and thus improve the quantity, quality, and consistency of the data collected. Also, implementing handhelds for surveying would allow the NHA to save time coding and analyzing their data sets and potentially collect more information on the floor.
• Failure to collect basic information such as zip codes and basic demographic data is a severe limitation that needs to be addressed for future marketing and funding opportunities. These were seen through small data sets and uncompleted observational demographic questions on surveys. Luckily, these problems can be fixed if proper training is given to all staff and more effort is put into data collection.

Based on the results of the coding and analysis conducted by the project team the following recommendations have been presented to the staff at the NHA:

Overall Recommendations:

• Continue to conduct evaluations in the museum since so much useful information can be obtained from them that will allow the NHA to enhance its programs, marketing, and ability to garner funding.
• Staff should be trained on all aspects of the museum’s evaluation process in order to improve the quality and quantity of the data that can be collected and reinforce the sense among staff that visitor evaluation is important.
• Create tailored survey instruments that always try to balance marketing needs with curatorial and programmatic needs, though inevitably will want some instruments with a heavier emphasis on one or the other depending on the point of the particular survey and needs of the museum at the time.
• Do not lose focus on the strengths of the museum such as the staff’s alignment with the four tenets of the NHA: programs, collections and properties, audiences and access, and organizational effectiveness.

Technology Recommendations:

• Use online survey generators for survey delivery since they can cut down the time it takes for coding and analysis with the use of handhelds or online surveying.
• Utilize handheld devices such as iPads© for portable surveying, since data collection and coding would be instantaneous and visitors may prefer to participate in interactive surveying.
• Utilize point of sale software to collect zip codes, membership information, and email addresses since every visitor must go through this part of the museum. This makes the point of sale the easiest location to collect random data samples.

Front-End Evaluation Recommendations:

• Utilize a front-end evaluation plan for future exhibits, programs or activities in order to create products that are engaging for visitors but also correspond with the museum’s mission.

Data Collection Recommendations:

• Make sure those who collect data write clearly and legibly so anyone could pick up their work and code or analyze the data.
• Use premade templates for inputting data into Excel© to save the time it takes to create these sheets and all data is organized in the same fashion.

Supplemental Material Recommendations:

• Be more economical about survey and other material creation for cost and labor efficiency.
• Provide a map with key locations and other pertinent museum and tour information so visitors can return to sites. This can also act as a marketing tool and that can be sold for revenue.
• Create a supplemental reading list for more information about tour topics. Have supplemental reading materials available at the end of the tour or at the NHA gift shop.

Using these recommendations, the staff will be able to gather more data from visitors and learn about their interests and opinions of the museum and its programs which can be used to enhance its programs, marketing, and funding.