Chapter 4. Practical Framework for Evaluating Traffic Safety Campaigns

The core deliverables developed for this project provide a practical framework for supporting SHSOs and other end users in selecting, developing, and implementing methods for evaluating traffic safety messaging campaigns. The guide is designed to walk users through the critical first steps of planning an evaluation based on key considerations to help users identify appropriate evaluation methods with the accompanying easy-to-use interactive Costing Tool indicating the extent of the required resources. The Costing Tool provides SHSOs and other end users with an impression of typical costs for applying each of the evaluation methods. It includes steps and activities for each type of evaluation, roles needed to complete the evaluations, hours for each activity, and resource estimations of other direct costs that may fluctuate based location, timing, the type of evaluation, the number of respondents/participants, or other inputs. Given this variation, the tool is not intended to be an exhaustive list of costs but rather provides ranges for each type of cost and assumes that SHSOs will research more realistic and timely local costs for each activity. The tool includes input fields for SHSOs to add those local costs to the spreadsheet, which will allow it to calculate total figures and provide budgets that are more applicable to a given campaign evaluation and local circumstances. The guide includes descriptions and explanations of the various elements included in the Costing Tool, as well as key considerations for selecting an evaluation method and other critical steps when planning a campaign evaluation. Additionally, the toolkit includes a Quick Reference Guide providing a condensed snapshot of the content for use with the Costing Tool). Collectively, these products are intended to help SHSOs select appropriate data collection methods based not only on cost but also the measures of interest that they select for their campaign evaluation.

Development of Toolkit Materials

To create realistic and scalable estimates for the Costing Tool, the team relied on their experience and expertise with the various data collection methods or consulted external entities to obtain costing information for methods they were not as experienced with. For example, several media strategy consulting vendors were consulted to identify typical costs for obtaining media impression data (e.g., gross rating points), web data (e.g., unique visits, clicks, referrals, etc.), and social media data (e.g., tweets, posts, engagement). NORC is an industry leader in qualitative research; thus, they tailored their established pricing plan to assist in the development of a Costing Tool for focus groups and interviews that includes the time and labor needed for instrument development, recruitment (including recruiting fees, where applicable), virtual or in-person data collection, and various levels of or approaches to qualitative data analysis. Similarly, NORC leveraged their experience with multi-modal survey data collection to identify the costs and capacity needed for programming web-based surveys, designing mailed paper-and-pencil surveys, and/or training phone interviewers. Estimated costs associated with survey deployment and administration include typical printing, recruitment, incentives, and mail costs; costs associated with logistics support for data entry based on ongoing work with vendors for their frequent national surveys; and the time and statistical expertise needed for different types of

analyses. Virginia Tech Transportation Institute (VTTI) used their extensive knowledge and expertise with on-road data collection methods to estimate the resources and tools required to support the construct and deployment of naturalistic driving studies at the state and local levels. Prior experience using citation and crash data also allowed VTTI to develop estimates of the resources required to obtain meaningful data.

The Toolkit Materials focus on the primary evaluation methods, including media impression data, focus groups, survey data collection, observational studies, naturalistic driving studies, and epidemiological studies. See the Guide for the Costing Tool and the Quick Reference Guide.

Limitations

A major limitation of the study was the inability of the team to secure subject matter experts and end users (i.e., SHSO personnel) to review the Toolkit Materials. Multiple attempts were made to enlist reviewers; however, existing workload and prior commitments were given as reasons for refusal by the overwhelming majority of those contacted. Although this may impact the potential usefulness of the toolkit due to lack of input and guidance from the intended end users, the team developed the Quick Reference Guide as an additional tool to assist with implementation of the framework and provide a very simple step-by-step guide to using the different worksheets of the Costing Tool.

Dissemination of Research Results

The primary audience for this research is SHSOs and local or private sector parties involved in the development and implementation of traffic safety campaigns. It is typically good practice to develop an evaluation plan for a traffic safety campaign while in the process of developing the actual campaign. Developing and refining the evaluation plan can act as a feedback loop to inform the development and design of the traffic safety campaign, including the intended audience and the goal of the campaign. Thus, the Toolkit Materials should be made available electronically to SHSOs and other interested parties via organizations such as the Governors Highway Safety Association or the Transportation Research Board so end users across the United States can begin to implement the materials. The materials are ready to use and easily disseminated by email or via the internet.