Chapter 2. Methods and Metrics to Evaluate the Effectiveness of Traffic Safety Campaigns

Foundational to the evaluation of traffic safety campaigns is an understanding of the methods that may be used to conduct campaign evaluations, along with the metrics and data associated with each method, and how these factors interact with one another and with the type and goal of specific traffic safety campaigns. Each evaluation method may address one or more levels of campaign targeting and outcomes; for example, while self-report data may reflect attitudinal changes, these data are less likely to provide reliable evidence of behavioral change due to response bias. Methods also vary regarding the resources and effort required for completion, the face-validity of the outcome, and the sample size required to draw conclusions.

The first step in evaluating campaign effectiveness is to define the success criteria and formulate evaluation questions that address those criteria, which require identifying the metrics to be measured. Of critical importance is a comprehensive understanding of what metrics exist, what information each metric contains, and how each metric maps to campaign outcomes (Table 1).

Table 1. Measurement metrics associated with intended campaign outcomes (Owens et al., 2023)

Measurement metrics are inextricably linked to methods of measurement, and different types of data (e.g., self-report data, media impressions data, observational data) can provide insight into the effectiveness of different messaging campaigns depending on the campaign outcomes. The scope of the campaign and available resources will also impact the selection of different methods of measurement, which include a range of resource and personnel investments, such as directed data collection like questionnaires and roadside surveys, evaluations of existing data sources like citation data, and novel data collection like naturalistic driving studies. Additionally, each method of measurement has its strengths and limitations, which will also be an important consideration when deciding on the evaluation method to use. The following describes the methods conducted for the literature review and environmental scan to identify recent traffic safety campaigns, along with the measurement metrics, methods of measurement, outreach modalities, and types of data associated with each intended campaign outcome.

Methods

Academic search engines (e.g., Google Scholar, Web of Science, and PsychINFO), and traffic safety and health communication journals were consulted, as well as technical reports, toolkits, government publications (e.g., US DOT), and organizational websites to identify current traffic safety campaigns and messaging.

Search Strategy.

Search parameters used different combinations of key words (Table 2) and applied Boolean operators (i.e., AND, OR) between search terms. Search strings were tailored to each source. For example, “AND” was applied for terms across columns (Campaigns AND Behaviors/Characteristics). In addition, “OR” was applied for terms within columns (Traffic

Safety Campaign OR Road Safety Campaign). The first 20 search results (titles and abstracts) were rapidly screened to identify articles of relevance to the literature review. If a relevant resource were identified in that set of results, the next 20 search results were reviewed. If another relevant resource was identified in the second set of results, then the next 20 results were reviewed, and so on until no additional relevant resources were identified. Traffic safety and health communication journal databases were also reviewed using key search terms.

Specific journals targeted include:

• Accident Analysis and Prevention
• Analytic Methods in Accident Research
• Traffic Injury Prevention
• International Association of Traffic and Safety Sciences Research
• International Journal of Injury Control and Safety Promotion
• Journal of Safety Research
• Safety
• Journal of Health Communication
• Journal of Transportation Safety and Security
• Journal of Transport and Health

Table 2. Search Terms Used to Identify Campaigns

Eligibility Criteria and Screening Process.

Article titles and abstracts were initially screened to determine merit for inclusion and ensure articles met the following criteria:

1. Published in the past 10 years (2013-2023)
2. English language
3. Focused on key words
4. Focused on road safety campaigns conducted in the U.S.

The full text of each article was reviewed to assess the relevance to traffic or road safety evaluation campaigns and any evaluation metrics and variables described. As a result of limited sources identified in the initial search, the eligibility criteria were broadened to encompass articles published between 1995 and 2012. This extension included the incorporation of injury prevention campaigns and health communication media campaigns into the search. The inclusion of these additional campaign types was intended to help capture pertinent media metrics for safety campaigns.

Data Extraction.

A matrix was created using Microsoft Excel to capture relevant information associated with each traffic safety campaign identified during the literature review and environmental scan. The team gathered information related to campaign type, location of the campaign, outcome of interest for the study, measurement metrics, measurement metrics description, measurement metrics definition, data collection methods, evaluation design, and theories.

Findings

There was a total of 40 sources that met the inclusion criteria. These sources covered a variety of traffic safety campaigns addressing different aspects of traffic safety, including, alcohol-impaired driving, distracted driving, bicycle and pedestrian safety, speeding, motorcycle safety, seat belt safety, and a combination of several types of campaigns.

The sources related to traffic safety campaigns included 21 peer-reviewed studies, 17 reports, a traffic safety web page, and a book chapter. The book chapter was added as a supplemental source not directly related to traffic or road safety campaign evaluations, but to health communications and media writ large, and was included to provide definitions of measures where those were missing from the literature or to offer insight into potential considerations for future safety campaigns. All traffic safety campaigns identified in the literature are presented in the accompanying Excel file (Evaluation Matrix), which provides the following details:

• The focus of the safety campaign and any associated context, including specific risky or protective behavior targeted by the campaign.
• The outcome of interest, such as the intended changes, impacts, or results that are measured or observed when evaluating campaign effectiveness.

• The type and format of campaign messages, including the number, frequency, periodicity, timing (primetime/drive time/daytime, etc.) variations in messages/formats, channels, and duration of campaign use of these channels.
• Measurement metrics and associated descriptions and definitions, including specific factors or indicators used when evaluating the outcomes or impact of the campaign. For example, key variables that are measured to evaluate the success, changes, or improvements that result from the campaign.
• The methods of measurement, which focuses on capturing details about the research methods, data collection instruments, survey tools, observational techniques, or other approaches used to gather information for evaluating campaign effectiveness. This includes both qualitative and quantitative measures.
• The design and scale of the campaign evaluation, such as the framework of the evaluation, sample size, number of surveys sent/received, etc.

Summary

The detailed breakdown of the methods and metrics used to evaluate the effectiveness of traffic safety campaigns is presented in the accompanying Excel file (Appendix A) with key points summarized below.

Mass media campaigns affect driver knowledge and awareness which are essential to many deterrence and prevention countermeasures. Changes in traffic knowledge is a measure that can be used to determine the effectiveness of road safety campaigns. Campaigns that utilized knowledge as a measure spanned the breadth of road safety topics, including bicycle and pedestrian safety, alcohol-impaired driving, seat belt safety, speeding, driving fatigue, and general road safety. These campaigns largely used questionnaires or surveys for data collection across multiple modes (i.e., phone, internet, and face-to-face) and were predominantly administered prior to the implementation of the education program or media campaign, and after the implementation of the program and campaign, to assess changes in knowledge.

Attitudes and perception metrics encompass a range of factors that influence an individual’s likelihood to engage in specific behaviors, shaping their opinions regarding these behaviors, and molding their perception of the associated risks. These metrics were applied in a wide array of the identified road safety campaigns, covering diverse topics such as alcohol-impaired driving, seat belt safety, speeding, driving fatigue, and general road safety. Data collection methods employed in these campaigns included questionnaires, surveys, interviews, focus groups, and self-reporting.

Campaign awareness was mentioned as a metric for measuring the effectiveness of varying road safety campaigns, including impaired driving campaigns. Participants’ awareness of the respective campaign was measured to better understand the reach and impact of these initiatives. Like mass media campaigns, these data were collected using questionnaires, surveys, and interviews. This also included formative or mid-campaign research, strategically aimed at evaluating and optimizing messaging strategies. In addition to campaign awareness, campaign

recognition was mentioned as a metric for evaluating campaign effectiveness. The difference between the two can be defined from a branding perspective. Brand (or in this case, campaign) recognition is the visual and audio cues people use to identify it, while brand awareness is knowledge that a brand or campaign exists. Campaign recognition, within this context, denotes an individual’s ability to recognize a campaign and where they had heard/seen it before. Data collection for assessing campaign recognition employs structured questionnaires, comprehensive surveys, in-depth interviews, and focused group discussions.

Media metrics were mentioned in the literature in relation to campaign evaluation, including measure units of messaging, earned media, website metrics, social media engagement and followers, reach, impressions, and cost. Examples of website metrics are the number of return visitors, average amount of time users spend on the site or on particular pages on the site, search terms that lead visitors to the site, number of downloads of materials hosted on the site, referrals to the site (i.e., which other sites with links to the site are visitors most commonly coming from) and number and destination of links out (in other words what sites are visitors moving on to next after visiting the site). Social media metrics include engagement counts or rate, which is a metric used to measure the level of interaction social media users have with a campaign, such as comments, likes, tweets or retweets, shares or reshares, reactions, and uses of campaign hashtags. Social media followers or measuring the rate/number of follower growth during a traffic safety campaign were also mentioned as an evaluation metric. Finally, reach and exposure are commonly used media metrics, with reach providing a measure of potential readership, listenership, or viewership, while exposure describes any time an audience member has viewed or heard campaign materials.

Changes in evaluation metrics related to crash data, such as traffic fatalities, fatality rates, and number of crashes, were used by a wide range of road safety campaigns, including those targeting motorcyclists, promoting seat belt use, focused on alcohol-impaired driving, or general road safety. Data supporting these measurement variables were collected through questionnaires, surveys, and interviews.

Thus, current practices to evaluate the effectiveness of traffic safety campaigns identified in the literature provide insight into the different evaluation methods and their associated metrics. To identify the most appropriate evaluation method, many factors need to be considered, and these vary depending on the campaign type and desired outcomes. Measurement metrics are inextricably linked to evaluation methods, and different types of data can provide insight into the effectiveness of different messaging campaigns.