Developing a Strategy to Evaluate the National Climate Assessment (2024)
Chapter: 5 Criteria for Choosing Audiences to Include in the Evaluation
5
Criteria for Choosing Audiences to Include in the Evaluation
As discussed in Chapters 3 and 4, there are a large number and variety of potential audiences of the National Climate Assessment (NCA). Many have highly specific and discrete needs that may not overlap with the needs of other audiences. These potential audiences may obtain information directly from the NCA, indirectly through an intermediary (perhaps without knowing that the NCA was the original source) and possibly with changes introduced by the intermediary, or information from sources other than the NCA. Some may not be examining information about climate change at all (e.g., they are too busy, uninformed, apathetic, or do not see the need). These factors greatly complicate the design of an evaluation.
There is no master list of all potential audiences, and, even if such a list existed, a random sample from that list would not be meaningful and generally would not contain enough people in each audience. Practically speaking, too, a data collection instrument that works well for one audience may not work well for another because of differences in needs and knowledge levels across the audiences. Thus, audiences will often need to be examined separately, and an evaluation will need to select a small number of audiences on which to focus attention. A staged approach, in which successive evaluations look at different audiences, can provide greater coverage than would be practical in a single evaluation.
This chapter provides general criteria for selecting audiences to include in an evaluation. The choice of audiences will be affected by the resources available for the evaluation, and there will be advantages and disadvantages for each audience that might be selected.
Selecting the audiences to assess in an evaluation of NCA use involves the analysis and consideration of the complexity and the multiple uses of the information provided by the NCA. The variety and range of applications of the information involve a diversity of users with different social, educational, geographical, and economic backgrounds, among other characteristics. The committee, in addressing this complexity, suggests that the designers of an NCA evaluation develop a logic model to visualize the NCA’s relationships across different audiences, uses of information, and impacts, and that they use network analysis to establish the relationships among multiple audiences. This recognizes that the NCA operates in a network of networks context, which provides evaluators the opportunity to identify gaps, including marginalized populations (audiences) whose needs may not have been addressed. Prioritizing those populations in planning an NCA evaluation can provide the U.S. Global Change Research Program (USGCRP) with insights to ultimately help address gaps and find opportunities to promote social justice, equity, and inclusivity. Setting priorities for audiences to involve in an evaluation will help in engaging those users of the NCA who play an important role in USGCRP’s logic model. The evaluation should include a focus on those whose decisions make a difference and those who play influential roles in the transmission of information from the NCA. This chapter articulates criteria for setting priorities, after discussing considerations that stem from the objectives of the evaluation.
AUDIENCES PROVIDING INFORMATION NEEDED FOR AN EVALUATION
In identifying criteria for selecting audiences, it is important to consider an evaluation as a whole, and what is needed is to gather perspectives from users and potential users. That means, in particular, that the selection of audiences accounts for the goal of supporting continuous improvement of the NCA. An evaluation is a project that is limited in resources and budget and reliant on the availability of audiences to participate or the information available about the audiences to be included. Criteria for prioritizing audiences for inclusion in an evaluation are discussed below, with particular attention to audiences that an evaluation will examine to understand the outcomes of the NCA.
Importance to USGCRP
The logic model should clarify for USGCRP which of its products’ users play significant roles in the pathways that achieve its goals. Those users should be included within the evaluation with enough breadth and depth to understand those pathways. Two audiences deserve special mention. The Global Change Research Act1 defines the NCA as a report to Congress and the president. It is accordingly logical for an evaluation design to include these two audiences among the priority populations to be engaged. As key audiences and users of the evaluation, it may also be useful to consult with them during the evaluation study design.
More generally, the logic model may suggest that a particular audience has a key role as a decision-maker or as a transmitter of information. To assess importance, USGCRP and the evaluator are encouraged to consider their hypotheses about which pathways will lead to the outcomes of greatest interest. Some users of the NCA make national policies, yet some policies of a limited scope may affect more choices on the ground, such as local urban planning rules to minimize flooding. Some outcomes may not focus on policy, such as raising awareness of climate impacts among underserved audiences and informing members of those groups of potential responses and resources for adaptation.
Some audiences are important because they act as intermediaries—nodes in the network of networks that transmit information to other nodes (see Chapter 4); examples include professional groups, educators, and news organizations. Chapter 6 provides examples of pathways where intermediaries play an important role.
Generalizability and Applicability
Any evaluation will be able to collect data from only a subset of potential audiences, and it is important to consider whether what is learned about each of these audiences and their use of the NCA can be applied more broadly. This is partly a matter of representativeness: are the audiences and uses investigated likely to be similar to other audiences and uses? Does the evaluation design collect information that provides insights that can be translated from those individuals who are included directly in the evaluation into an understanding relevant to at least part of the wider population? Generalizability is one consideration in determining priorities.
In setting priorities, it is useful to consider that the organizations already known to USGCRP may not be representative of the audience that the Program considers important to inform. Within a priority audience, the evaluator should seek to involve some respondents who have not already worked with the NCA, so as to minimize selection bias, as it is called in statistics.
As discussed in Chapter 6, however, the evaluation question being investigated affects how one considers generalizability. From a statistical perspective, generalizability refers to the applicability of the statistic to a population beyond the respondents. More generally, one might consider whether an evaluation finding has relevance to a broader population or broader class of events, even if a statistic cannot be considered to be representative. If the question is what percentage of an audience is aware of the NCA, and the method used is a question in a survey, one would want to know whether the survey estimate reflects the entire audience and perhaps the degree to which the statistic applies to other audiences.
___________________
1 Global Change Research Act of 1990, 15 U.S.C. Chapter 56, Public Law 101-606, 104 Stat. 3096-3104.
If the question is how the NCA is used in a particular decision context, a case study may be informative, even if the narrative it produces is not readily applicable in other situations. In particular, case studies of unusual success may offer insights relevant to the logic model, as well as suggesting lessons learned that might assist in replicating their success. Similarly, a finding that the online NCA report is difficult for some to navigate for a particular reason can be useful whether or not one is able to estimate the number who would benefit from a change.
When focusing on audiences, generalizability is defined in terms of the ability to generalize to a particular audience, not to the overall population. For example, suppose that an evaluation included a case study of a particular nongovernmental organization (NGO), such as the Union of Concerned Scientists (discussed in Chapter 6). At the most basic level, one would want to know the degree to which the case study covered the entire organization, or only a few individuals within it. It is not necessary for the results to be generalizable to the full group in order to be useful, but this is one factor to consider. Next, one might ask whether at least some of the findings would apply to other NGOs as well. Few results can be generalized to describe the entire universe of NGOs, because NGOs collectively are so diverse in their purposes, structures, and operations. There would be no realistic way of generalizing still further to the entire United States because the nation is much more than and much different from an aggregation of NGOs. Nonetheless, one may still find principles that are broadly applicable, such as that some audiences might need a much higher level of specificity—such as localization—than is contained in the NCA. As a corollary, the questions to be addressed sometimes will be audience specific, and an evaluation will not apply the same questions to every audience; for example, some audiences may be users only and not disseminators of NCA information.
A related practical consideration relates to sampling (Lance and Hattori, 2016; Palinkas et al., 2015). To understand the extent to which a phenomenon exists—for example, what proportion of local health department officials are familiar with the NCA—the evaluators would seek to survey all or a representative sample of those health officials. The feasibility of conducting a survey that will produce generalizable results about all U.S. local health departments depends on (1) how feasible it is to gather a list of all local health departments (the universe of local health departments); and (2) the ability to field a survey with adequate response rates that reflect that full universe. (As described in Chapter 6, one strategy for addressing these concerns is adding questions to existing surveys that already have well-established sampling processes and robust response rates.)
Diversity
One aspect of evaluation coverage deserves special attention: the inclusion of marginalized or underserved audiences. The needs of some populations have not been considered consistently in the development or provision of climate services, and they may lack access to the NCA or to decision-makers. Engaging their perspectives and understanding their needs could make the NCA a much stronger product. In gauging the size of this population, it is important to consider the definition of underserved. Many may not be aware of the NCA but still use its information or are affected by it. To the extent that climate information traceable to the NCA is reflected in public policies—such as development guidelines for floodplains or in planning for cooling centers during heatwaves—the NCA may be said to touch some of the decision-making and experiences of a wide range of residents of the United States. The specification of what counts as use of the NCA will affect the population estimated to be underserved. Such a definition is one that USGCRP could articulate as it develops its logic model.
Drawing upon research in environmental justice that identifies communities and demographic groups that lack access to climate services or climate information (Dolšak and Prakash, 2022; Tripati et al., 2024), the evaluator could propose an evaluation instrument to illuminate how climate information reaches and is used by underserved groups. As discussed in Chapter 4, a mapping of the network along which the NCA is shared can identify key nodes; the persons at those nodes are likely to have information about the audiences they are trying to reach and the limitations they encounter in doing so. As noted earlier, information gathered in this way relates to only a portion of those who do not use the NCA.
In addition, some people, including climate skeptics, do not believe that federal climate science is useful. Better understanding this population could provide information useful in improving the NCA and climate services in the future. In a recent study, the Pew Research Center interviewed climate skeptics and observed the way that
mistrust of the news media interacted with reluctance to believe in climate change (Pasquini et al., 2023). Those who are not well served by existing climate services programs may also be affected by their access to, and trust in, news media; these and other interaction effects could be studied using small groups.
The Pew Research Center’s studies indicate that climate change is perceived to be an important issue among U.S. residents, but one not as pressing as other problems, such as the state of the economy or health care (Tyson et al., 2023). Moreover, opinions about climate change are polarized, with Democrats readier to believe that climate change is important enough to warrant action by government, while Republicans are more cautious in their support. Substantial majorities do support climate mitigation policies, however, such as the development of renewable energy (Tyson et al., 2023).
What is more germane to the NCA is the modest confidence of the public in climate scientists. Pasquini and Kennedy (2023) introduce their article by saying:
Only about one-third of Americans think climate scientists understand very well whether climate change is happening. . . . And only about a quarter or less say climate scientists understand very well the effect climate change has on extreme weather, its causes and the best ways to address it.
These findings suggest that many would not find the NCA useful even if they knew of its existence.
An evaluation exploring these beliefs would likely clarify what is expected of the NCA, as well as prompt ideas about how climate information could be more useful to selected audiences. The committee recognizes, however, that there are serious methodological issues in trying to conduct such a study, both in identifying the audiences to be studied and in judging how representative the results would be. What may be feasible is to conduct case studies or focus groups with a limited number of people in such audiences to better understand what information they have and what difficulties they face.
There is another population of interest: those who are aware of the work of USGCRP but do not find the NCA and associated products useful. Such people may be difficult to identify; it is easier to identify users (e.g., through monitoring downloads or citations) than nonusers. For these reasons, it may not be practical to specifically target nonusers. Instead, one might target groups based on their potential for benefiting from the NCA (e.g., target urban planners in general rather than urban planners who make use of the NCA), and then ask about their knowledge of and interest in climate science information, and their sources of information. Since knowledge and use of the NCA and its products might depend on the prevalence of climate services (provided by the federal government or other groups), useful information might also be gained by comparing individuals within a given community of practice (such as urban planners or public health officials) who do have ready access to climate services with those in that community who do not. Such a comparison might help in identifying not only how climate services have promoted use of NCA-derived information but also groups for which outreach might be particularly useful.
FEASIBILITY OF EXAMINING PARTICULAR AUDIENCES
In setting priorities for audiences and methods to apply in an evaluation, practical considerations such as identifiability, data availability, accessibility, and cost affect the options for evaluation design, and for how an evaluation is conducted or phased.
Identifiability.
Although Congress, the president, and federal agencies have known and clearly identifiable memberships, other important audiences do not. Consider that, among others, there are many NGOs, only some of which are involved in climate change; there are many educators, but there is no national list to sample from and no way to screen them by their relationships to climate science information; and there are diverse media, ranging from traditional outlets to a wide variety of blogs and other sources on the internet. In those cases, generalizability of evaluation findings does not mean representativeness in a statistical sense. It will not always be feasible or practical to develop a comprehensive list or to generate statistics that can be considered representative; instead, one may want to focus on a particular subaudience (such as a specific NGO) and seek findings that may be applicable more broadly.
Data availability.
Some data may be readily accessible to an audience that is being studied, such as the amount budgeted for climate-related activities or the number of individuals within an organization engaged in climate-related activities. In other cases, data may be available only with significant effort or not at all. These considerations affect which audiences to engage, what information might reasonably be obtained from them, and how reliable the information provided may be. An evaluator compiling secondary data (existing information that was not collected for the purpose of the evaluation) will need to assess the quality of those data and their relevance to the overarching evaluation questions.
Accessibility.
Members of some audiences are less accessible than others. Interviews may be difficult to arrange, permission to reach out to subjects may not be obtainable, and securing the return of surveys or other measurement instruments may be uneven or impossible.
Cost.
Budgetary and other resource limitations may make desirable audiences difficult to engage, forcing trade-offs between audiences or reductions in sample sizes.
EVALUATION STAGING
As is implied by the history of the NCA (Chapter 2), the broad relevance of the climate science in the NCA for decision-making (Chapter 3), and the complexity of the network of networks (Chapter 4 and considerations above), setting priorities to select audiences to include in the evaluation needs to engage with USGCRP’s goals. It is a practical necessity to proceed in stages.
The first stage is an evaluability assessment, discussed in Chapter 7. That includes creating an initial roadmap of the networks believed to be most important to achieving the Program’s purposes. In mapping the NCA’s network of networks, using low-cost methods such as citation analysis complemented by case studies, the objective is to clarify the logic model and the evaluation questions to be answered. In particular, network analysis is likely to identify structural holes: portions of the NCA network that are connected to one another by only a few intermediary connections. These intermediary nodes, and the way that they provide connections, may play a significant role in the spread of information from the NCA. Some of these may be worth highlighting in the logic model with evaluation questions developed to examine the role of these boundary-spanning nodes. Chapter 7 discusses this initial reconnaissance in greater detail.
Insights from an initial mapping can then be combined with the considerations discussed in this chapter, in order to choose a set of audiences to include in an evaluation. Those audiences will need to have the appropriate breadth and depth to answer the evaluation questions, while also being practical to work with. In developing a roadmap, an evaluator needs to be aware of the way the methods used in the reconnaissance provide only a limited view of the network of networks; as discussed above, exploring why potential audiences do not use the NCA requires methods that engage audiences unconnected to the network of networks. The initial roadmap is only a starting point that provides information to pose questions to audiences in a way that can provide reliable answers to important evaluation questions.
CONCLUSIONS AND RECOMMENDATIONS
The discussion above can be summarized in the following conclusions and recommendations, which set the stage for the discussion of methods in Chapter 6. Each conclusion pertains to a criterion for prioritizing audiences:
Conclusion 5-1 (Importance to USGCRP): The legislative mandate, logic model, and evaluation questions can be used to designate key audiences.
Conclusion 5-2 (Decision-making): (a) Where the logic model focuses on outcomes of national scope, audiences that influence or make decisions at national scale are the most relevant. (b) Where the logic model focuses on outcomes that matter because they result from many decisions made in a decentralized fashion, audiences are likely to be numerous and diverse.
Conclusion 5-3 (Information transmission): Some audiences may be prioritized for inclusion in an evaluation because they are hypothesized to be key nodes in transmitting climate information. Existing communications networks such as the mass media, as well as communications within professional organizations or communities may deserve priority where they play influential roles in the logic model.
Conclusion 5-4 (Generalizability): When feasible, there is value in choosing audiences for whom the results would be generalizable (i.e., the evaluation findings for an audience or subgroup within an audience are likely to be applicable to other audiences that are similar in a relevant dimension). Generalizability is not always feasible depending on the nature of the audience.
Conclusion 5-5 (Matching audiences to questions): In many instances, audiences need to be matched to the question to be investigated and comprehensive coverage or representative sampling may not be relevant or feasible.
Conclusion 5-6 (Diversity): Targeting audiences that are collectively diverse is important for two reasons: (1) to provide a more complete picture of how the NCA is or is not used, and (2) as a matter of equity, some marginalized audiences may lack access to the NCA or to decision-makers, but consideration of their needs can make the NCA a stronger product.
Conclusion 5-7 (Feasibility): Because of cost and other practical limitations, not all audiences can be evaluated using the most powerful evaluation methods. Sometimes these audiences can be examined in a more limited but still useful way using different methodologies.
Recommendation 5-1: In choosing which groups to study as part of an evaluation, the U.S. Global Change Research Program should seek diversity (including a focus on marginalized populations) similar to that of the participants and audiences with which the Program seeks to engage.
Recommendation 5-2: The U.S. Global Change Research Program (USGCRP) should select audiences to include in evaluation based on the following criteria: importance in USGCRP’s logic model, including (1) the role of an audience in climate-related decision-making; (2) the role of an audience in the transmission of climate information for decision-making; (3) the generalizability of results from an audience to other populations; and (4) feasibility, diversity, and suitability for the evaluation question and method used. A targeted audience does not need to meet all of these criteria, but the audiences prioritized in an evaluation should meet these criteria collectively.
Recommendation 5-3: The U.S. Global Change Research Program (USGCRP) should continue to pay attention to how climate change affects historically marginalized communities and underrepresented audiences by continuing to provide pathways for including them as part of the development process of the NCA, making sure that they are heard throughout the development process—and more broadly by sustaining efforts to provide information about the climate-associated needs of those groups. The evaluators engaged by the Program should include persons whose backgrounds and lived experiences afford them understanding of underserved communities, including those exposed to the impacts of climate change.
Recommendation 5-4: Guided by its logic model, the U.S. Global Change Research Program (USGCRP) should progressively develop a roadmap of the network of networks in which the NCA and its related products are used. This roadmap would show the nodes that are hypothesized to play key roles in the diffusion of usable knowledge in the NCA and related products. The results should be considered when selecting which audiences to target in the evaluation.
