Digital Visitor Evaluations at the Nantucket Historical Association

Sponsor: Nantucket Historical Association

The NHA Group: (left to right) Catherine Bonner, Adam Karcs, and Emily Perry

The NHA Group (left to right): Catherine Bonner, Adam Karcs, and Emily Perry
Sponsor Liaison: Katie Schoorl, Manager of Visitor Operations
Marjan Shirzad, Director of Visitor Experience
Student
Team:		 Catherine Bonner, Adam Karcs, Emily Perry
Abstract: The goal of this project was to create technology-assisted surveys for the Nantucket Historical Association (NHA). After researching industry best practices and consulting with NHA staff, the student team used web-based software to create a 2014 NHA Programs Survey and Museum Survey. The team conducted public testing to gauge impressions of the digital technology, and analyzed response data from the implemented surveys. With the instruments developed, the NHA will be able to collect more in-depth visitor feedback that will help the NHA improve its administrative decision-making and adjust its practices to better meet its patrons’ needs. It is our hope that the NHA continues to use and create digital surveys to further enhance visitor experiences.
Links: NHA_Final_IQP_Report
NHA Presentation

Executive Summary

In recent decades, the museum industry has taken an increasingly constructivist approach to its educational practices, focusing more on mediating visitor interpretation rather than providing structured knowledge in a didactic fashion. The importance placed on the visitors’ personal interpretation of exhibits has led to an increased need for comprehensive visitor evaluations. The feedback collected through evaluation contributes significantly to all areas of museum operation, such as marketing, collections management, exhibit design, and visitor services. The implementation of these types of systematic evaluation, however, often requires substantial human resources. Progressively, museums are integrating technology into visitor evaluation practices to increase efficiency and to facilitate the process of gathering visitor feedback, significantly reducing the need for human supervision.

While the museum industry as a whole is incorporating technology into its culture, the focus of this project is on the operations of the Nantucket Historical Association—specifically the Nantucket Whaling Museum. Like other small museums, the Whaling Museum has limited staff, time, and resources—making the usage of pen and paper surveys mostly infeasible.  Technology has the potential to alleviate staff members of many of the responsibilities regarding the facilitation of visitor surveys. The student team used digital technologies to make the Nantucket Whaling Museum’s surveying methods as streamlined and comprehensive as possible.

 

Statement of Project Goals and Objectives

The overall goal of this project was to explore how digital visitor evaluations could be implemented at the Nantucket Historical Association. In order to achieve this goal, the team established four project objectives as described below, each with a set of associated tasks.

  • Objective 1: Determined state-of-the art surveying practices in the museum industry;
  • Objective 2: Examined the current and preferred visitor evaluation methods at the Nantucket Whaling Museum;
  • Objective 3: Developed a suite of technology-assisted survey instruments; and,
  • Objective 4: Provided recommendations for the use and maintenance of the instruments developed.

 

Methods

          To accomplish the project goal, the team began by interviewing NHA staff members to gather the opinions of different departments, their respective goals, and more details on the data they currently collected. This determined what surveys—and what distribution methods— best suited each of the museum’s needs. The team categorized the responses and developed questions to be included in each survey. Ultimately, the team narrowed their focus to two surveys: the 2014 NHA Programs Survey and the Museum Survey.

As the team developed survey questions, they simultaneously created digital versions using an online survey provider. In an iterative process, the team engaged the museum staff to refine the content and digital format of each survey until both the student team and the staff were satisfied. After multiple rounds of in-house testing, the 2014 NHA Programs Survey was sent out to the NHA contact list and the Museum Survey had undergone three rounds of public testing in the museum. After analyzing the results of the two surveys, the team provided recommendations for the maintenance of the instruments as well as ways the NHA can further enhance their visitor evaluation practices.

 

Findings

Objective 1: Industry Practices

To determine which online survey provider was the best fit for the NHA, the team utilized free trials, phone consultations, and online demonstrations. Conversations with survey providers centered on particular needs of the NHA, such as: the annual expected number of responses, the anticipated modes of survey delivery, and desired design elements. After receiving pricing quotes and trial accounts, the team recommended that the NHA use SurveyGizmo’s Basic package to create future surveys. The NHA purchased a full annual subscription to SurveyGizmo, and the student team created surveys using this online account.

 

Objective 2: Current and Preferred Practices

Through conducting interviews with NHA staff members, the major goals and areas of interest the team identified were advertising, branding, pricing, expansion of online ticketing, and email database expansion. The primary interests of our sponsor determined what surveys—and what distribution methods—best suited each of the museum’s needs. Ultimately, the team narrowed their focus to two surveys: the 2014 NHA Programs Survey and the Museum Survey. The 2014 NHA Programs Survey was an instrument to collect program attendees’ opinions on the previous year’s public programs, as well as generate feedback to guide the next year’s programming. The Museum Survey was intended to serve as an in-house exit survey for visitors to the Whaling Museum. This would collect useful information about the visiting body, as well as feedback regarding the museum’s exhibitions, galleries, and programs.

 

Objective 3: Develop Surveys

The team explored all the features offered within the Basic package of SurveyGizmo, and began developing the surveys in their digitized forms. The team utilized important features such as question routing and theme customization to create engaging and visually attractive surveys. Throughout the process, the team found ways to address the different interests of the NHA outlined above.

Then, alongside NHA staff, the team altered the wording, order, and flow of the survey questions many times to make the survey instrument more inviting to the visitor. The sponsors’ perspectives and experience contributed numerous useful modifications to the surveys, and in many ways created a more humanistic experience in line with their brand as a modern and friendly historical association.

After many iterations of both surveys, the team was able to produce thoroughly tested deliverables. The 2014 NHA Programs Survey was sent to the entirety of the NHA’s contact database—approximately 5,000 people—via email. The Museum Survey was administered to visitors on iPad minis throughout the Whaling Museum over three days of testing.

The 2014 NHA Programs Survey received a total of 59 completed responses from December 2nd to December 8th. This is not a large enough sample size to extrapolate results to the entire population of the NHA’s mailing list, and was a lower-than-expected response rate. However, the team still performed analysis of these results via Excel and SurveyGizmo’s Summary Report feature.

During public testing of the Museum Survey, the student team asked each survey taker a series of follow-up questions regarding usability of the instrument. This feedback added helpful third-party perspectives, and contributed to important modifications of the survey. Overall, the team encountered less resistance from visitors than expected. Visitors generally responded positively to the digital prototype, and appeared interested and engaged while using it.

 

Conclusions and Recommendations

In-house Museum Survey

Conclusion 1: The qualitative results of our public testing of the Museum Survey indicate that visitors to the Nantucket Whaling Museum are not opposed to the idea of completing a survey on a small touchscreen device. Furthermore, we found that many of Nantucket’s visitors and residents are pleased to provide their feedback on the NHA and its programs and exhibits.

Recommendations: To ease burdens on staff, we recommend the NHA keeps two iPad minis on stands. We also suggest that the Whaling Museum utilize SurveyGizmo’s “Kiosk Mode” feature which automatically refreshes the survey if it is left unattended for a certain amount of time.

Conclusion 2: During our testing, team members were able to stand near the survey taker as they used the devices. It is reasonable to conclude that in a different environment—where the survey taker would not feel as closely watched—there is a greater chance that users will try to exit out of the survey and open other applications.

Recommendation: As a precaution, we recommend the use of Guided Access on all iPad minis used for the Museum Survey which will provide limitations to the visitors so that they stay within the survey, and do not power off the device.

 

Large Scale Survey Distribution

Conclusion: As shown by the distribution of the 2014 NHA Programs Survey, email is an effective tool to disseminate a survey to members of the community. Therefore, there is the potential to collect large amounts of data via survey campaigns.

Recommendations: Given the low percentage of completed responses against the number of emails opened in the distribution of the 2014 NHA Programs Survey, we recommend the NHA send follow-up emails, social media blasts and offer incentives to encourage further responses.

 

Modifying and Creating New Surveys

Conclusion: In the future, the NHA will want to create different surveys for other purposes or because museum programs or activities have changed. Because all of the surveys are available online, all preliminary projects are preserved in the account, so that the NHA can use them as a starting point if so desired. These developed surveys consist of: Initial Membership Survey, Membership Renewal Survey, and Individual Program Survey.

Recommendation: Based on the performances of the 2014 NHA Programs Survey and the Museum Survey, it is our recommendation that the NHA implements the other preliminary surveys as well, and distributes them via the appropriate avenues of communication.

 

Using Survey Data

Conclusions: From our analysis of responses to the 2014 NHA Programs Survey, we have found that surveys provide useful feedback that should not be ignored. These comments effectively enable the museum to have an honest “conversation” with its patrons through the digital medium.

Recommendations: For the organization of this response data, we recommend the use of Summary Reports built in to SurveyGizmo. Summary reports provide high-level analysis—including charts and graphs—that are sufficient enough for most of the NHA’s needs. For more in-depth analysis, the team recommends importing data from the 2014 NHA Programs Survey into the Excel workbook developed by the team. The workbook created collapses data and makes sorting and filtering possible.

 

Afterword

            This project successfully provided the NHA with a foundation for further use of digital technology in its surveying practices. The instruments delivered will be useful tools for guiding future executive decisions regarding different aspects of museum and program management. Most importantly, these surveys enhance the organization’s ability to have more open and honest conversations with its patrons through the digital medium—thereby allowing more members of the community to express their concerns with, and regards for, the significant efforts of the Nantucket Historical Association.