NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001

This activity was supported by a contract between the National Academy of Sciences and the National Aeronautics and Space Administration. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.

International Standard Book Number-13: 978-0-309-72500-2
International Standard Book Number-10: 0-309-72500-3
Digital Object Identifier: https://doi.org/10.17226/27923
Library of Congress Control Number: 2024949739

This publication is available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Copyright 2024 by the National Academy of Sciences. National Academies of Sciences, Engineering, and Medicine and National Academies Press and the graphical logos for each are all trademarks of the National Academy of Sciences. All rights reserved.

Printed in the United States of America.

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2024. Developing a Strategy to Evaluate the National Climate Assessment. Washington, DC: The National Academies Press. https://doi.org/10.17226/27923.

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.

The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president.

The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.

The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.

Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.

Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.

Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.

Rapid Expert Consultations published by the National Academies of Sciences, Engineering, and Medicine are authored by subject-matter experts on narrowly focused topics that can be supported by a body of evidence. The discussions contained in rapid expert consultations are considered those of the authors and do not contain policy recommendations. Rapid expert consultations are reviewed by the institution before release.

For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.

COMMITTEE TO DEVELOP A STRATEGY TO EVALUATE THE NATIONAL CLIMATE ASSESSMENT¹

KAI N. LEE (Chair), Principal, Owl of Minerva, LLC

ANN M. GALLAGHER, Science Education Coordinator, U.S. National Park Service; Ph.D. Candidate, Strome College of Business, School of Public Service, Old Dominion University

MATTHEW O. GRIBBLE, Associate Professor and Associate Chief for Research in Occupational, Environmental, and Climate Medicine, University of California, San Francisco

SCOTT KALAFATIS, Deputy University Director, Northwest Climate Adaptation Science Center

JESSICA KRONSTADT, Program Director, Planetary Health Alliance

GLYNIS C. LOUGH, Affiliate, Aspen Global Change Institute

MICHELLE MIRO, Senior Information Scientist, RAND Corporation

ARIANE PINSON, Research Social Scientist, U.S. Army Corps of Engineers

UROOJ RAJA, Assistant Professor, Loyola University Chicago

CARLOS RODRIGUEZ FRANCO, Senior Forester, U.S. Forest Service

KATHLEEN SEGERSON, Board of Trustees Distinguished Professor of Economics, University of Connecticut

KRISTIN MARIE FISCHER TIMM, Research Assistant Professor, University of Alaska Fairbanks

Study Staff

STEVEN STICHTER, Study Director (as of March 25, 2024), Senior Program Officer, Board on Atmospheric Sciences and Climate (BASC)

BRADFORD CHANEY, Senior Program Officer, Committee on National Statistics

LINDSAY MOLLER, Senior Program Assistant, BASC

HUGH WALPOLE, Study Director, Associate Program Officer, BASC (until March 22, 2024)

BOARD ON ATMOSPHERIC SCIENCES AND CLIMATE

MARY GLACKIN (Chair), The Weather Company, an IBM Business (Retired)

JOSEPH ÁRVAI, University of Southern California

CYNTHIA S. ATHERTON, Heising-Simons Foundation

ELIZABETH A. BARNES, Colorado State University

BRAD R. COLMAN, The Climate Corporation (Retired)

BART E. CROES, California Air Resources Board (Retired)

MINGHUI DIAO, San Jose State University

NEIL DONAHUE, Carnegie Mellon University

LESLEY-ANN DUPIGNY-GIROUX, University of Vermont

EFI FOUFOULA-GEORGIOU (NAE), University of California, Irvine

KEVIN GURNEY, Northern Arizona University

ANDREA LOPEZ LANG, University of Albany

MARIA CARMEN LEMOS (NAS), University of Michigan

ZHANQING LI, University of Maryland

AMY MCGOVERN, Oklahoma State University

LINDA O. MEARNS, National Center for Atmospheric Research

JONATHAN A. PATZ (NAM), University of Wisconsin-Madison

KEVIN REED, Stony Brook University

J. MARSHALL SHEPHERD (NAS/NAE), University of Georgia

ARADHNA TRIPATI, University of California, Los Angeles

BERNADETTE WOODS PLACKY, Climate Central

National Academies Staff

ELIZABETH EIDE, Acting Director

MAGGIE WALSER, Acting Director

KATELYN CREWS, Program Assistant

APURVA DAVE, Senior Program Officer

MORGAN DISBROW-MONZ, Program Officer

KATRINA HUI, Associate Program Officer

ANNIE MANVILLE, Program Assistant

BRIDGET McGOVERN, Program Officer

APRIL MELVIN, Senior Program Officer

LINDSAY MOLLER, Senior Program Assistant

RACHEL SILVERN, Program Officer

STEVEN STICHTER, Senior Program Officer

COMMITTEE ON NATIONAL STATISTICS

KATHARINE G. ABRAHAM (Chair), Department of Economics, University of Maryland, College Park

MICK P. COUPER, Institute for Social Research, University of Michigan

WILLIAM “SANDY” DARITY, Sanford School of Public Policy, Duke University

ROBERT M. GOERGE, NORC at the University of Chicago

ERICA L. GROSHEN, School of Industrial and Labor Relations, Cornell University

DANIEL E. HO, Stanford Law School, Stanford University

HILARY W. HOYNES, Goldman School of Public Policy, University of California, Berkeley

H. V. JAGADISH, Michigan Institute for Data Science, University of Michigan

SHARON LOHR, School of Mathematical and Statistical Sciences, Arizona State University, Emerita

NELA RICHARDSON, ADP Research Institute, Roseland, NJ

ELIZABETH A. STUART, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health

National Academies Staff

MELISSA CHIU, Senior Board Director

CONNIE CITRO, Senior Scholar

BRADFORD CHANEY, Senior Program Officer

DANIEL CORK, Senior Program Officer

ALEX HENDERSON, Senior Program Assistant

DAVID JOHNSON, Senior Program Officer

REBECCA KRONE, Administrative Coordinator

CHRIS MACKIE, Senior Program Officer

ANTHONY MANN, Program Associate

KRISZTINA MARTON, Senior Program Officer

JENNIFER PARK, Senior Program Officer

SITARA RAHIAB, Senior Program Assistant

KATRINA STONE, Senior Program Officer

BOARD ON ENVIRONMENTAL CHANGE AND SOCIETY

MICHAEL P. VANDENBERGH (Chair), Vanderbilt University

BILAL M. AYYUB, University of Maryland, College Park

EDUARDO S. BRONDIZIO, Indiana University Bloomington

LISA DILLING, University of Colorado Boulder

KENNETH GILLINGHAM, Yale University

MARY H. HAYDEN, University of Colorado

LORI HUNTER, University of Colorado Boulder

STEPHEN H. LINDER, University of Texas Health Science Center at Houston

GLEN M. MACDONALD, University of California, Los Angeles

GARY E. MACHLIS, Clemson University

BENJAMIN PRESTON, RAND Corporation

JACKIE QATALIÑA SCHAEFFER, Alaska Native Tribal Health Consortium

JESSE C. RIBOT, American University

MADELINE I. SCHOMBURG, Energy Futures Initiative Foundation

BENJAMIN K. SOVACOOL, Boston University

ADELLE DAWN THOMAS, University of the Bahamas

CATHY L. WHITLOCK (NAS), Montana State University, Emerita

National Academies Staff

PATTI SIMON, Acting Director

JOSHUA LANG, Program Coordinator

JOHN BEN SOILEAU, Program Officer

DANIEL TALMAGE, Program Officer

Reviewers

This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published report as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.

We thank the following individuals for their review of this report:

THOMAS DIETZ (NAS), Michigan State University (retired), Grand Isle, Vermont

KENNETH FRANK, Michigan State University, East Lansing

JOY FRECHTLING, Westat, Rockville, Maryland

CHRISTINE KIRCHHOFF, The Pennsylvania State University, University Park

GARY YOHE, Wesleyan University, Portland, Connecticut

Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations of this report, nor did they see the final draft before its release. The review of this report was overseen by KRISTIE L. EBI, University of Washington, Seattle, and CHRISTOPHER B. FIELD (NAS), Stanford University, Stanford, California. They were responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies.

Contents

Key Terms Used in This Report

SUMMARY

1 INTRODUCTION

Evaluations

Statement of Task

The Committee’s Approach

Outline of the Report

2 BACKGROUND OF THE NATIONAL CLIMATE ASSESSMENT

Goals of the NCA

Process Used to Create the NCA

Broadening Audiences and Goals

Past Evaluations of the NCA

Findings, Conclusions, and Recommendations

3 FRAMEWORK OF AN EVALUATION

Overview of Evaluation

Conclusions and Recommendations

4 NETWORK ANALYSIS AND A NETWORK OF NETWORKS

Network Analysis

Some Concepts and Uses of Network Analysis

Applying Network Analysis to Evaluation

Findings, Conclusions, and Recommendations

5 CRITERIA FOR CHOOSING AUDIENCES TO INCLUDE IN THE EVALUATION

Audiences Providing Information Needed for an Evaluation

Feasibility of Examining Particular Audiences

Evaluation Staging

Conclusions and Recommendations

6 METHODOLOGIES AND APPLICATIONS TO PARTICULAR AUDIENCES

Overview of Potential Methodologies

Illustrative Applications to Particular Audiences

Conclusions and Recommendations

7 IMPLEMENTING THE EVALUATION FOR CONTINUOUS IMPROVEMENT

Timing and Sequencing of Evaluation Work

Communication of Evaluation Findings

Legal and Process Considerations

Choosing a Contractor

Conclusions and Recommendations

8 PUTTING IT ALL TOGETHER

Asking the Right Questions

Dealing with Multiple Audiences

Choosing Appropriate Methodologies

Continuous Evaluation and Improvement

Multistep Approach to Evaluation

REFERENCES

APPENDIX A: COMMITTEE MEMBERS

APPENDIX B: NCANET

APPENDIX C: CROSSWALK BETWEEN STATEMENT OF TASK AND OVERARCHING EVALUATION QUESTIONS

APPENDIX D: NETWORK ANALYSIS: ADDITIONAL MEASURES AND STRATEGIES

APPENDIX E: DERIVATIVE PRODUCTS

APPENDIX F: FURTHER INFORMATION ABOUT FEDERAL PROGRAMS

Boxes, Figures, and Tables

BOXES

S-1 Statement of Task

1-1 Statement of Task

2-1 Public Interest in Climate Change

4-1 Information from the National Climate Assessments Spreads Along Networks

4-2 Example Types of Relevant Connections for Evaluation

4-3 Illustrative Networks That Can Be Studied as Components of a Network of Networks Transmitting Knowledge from the NCA

FIGURES

S-1 Illustrative logic model for an evaluation of the National Climate Assessment (NCA)

2-1 Budget crosscut for the U.S. Global Change Research Program (USGCRP)

3-1 Attribution and contribution

3-2 Sample logic model

3-3 Illustrative logic model for an evaluation of the National Climate Assessment (NCA)

3-4 Sample user pathway through the logic model

4-1 Transmission of information from the National Climate Assessment (NCA)

8-1 Illustrative logic model for an evaluation of the National Climate Assessment (NCA)

TABLES

2-1 Expansion of National Climate Assessments (NCAs) over Time

3-1 Preliminary Overarching Evaluation Questions for the National Climate Assessment (NCA)

6-1 Selected Advantages and Disadvantages of Data Collection Methods

6-2 Data Collection Methods and Considerations for Overarching Evaluation Questions

C-1 Crosswalk Between Themes and Questions from the Committee’s Statement of Task and Preliminary Overarching Evaluation Questions Presented in Chapter 3

F-1 USDA Climate Change Hubs Accomplishments During 2013–2023

Key Terms Used in This Report

Audience The receiver or user of climate change information, including those affected by climate change. The Global Change Research Act (GCRA, 15 U.S.C. Chapter 56, Public Law 101-606, 104 Stat. 3096–3104) of 1990 requires the U.S. president and Congress to be audiences of the National Climate Assessment (NCA).

Case study An in-depth study, such as of one group or organization, that is intended not to provide statistically representative results but rather to allow a more thorough investigation than would be possible in a survey.

Cognitive interviews Interviews to determine the thought process that a person goes through to complete a task. They are often used for testing survey questionnaires (e.g., to find out how the respondent would interpret the question, whether the question has the right choices, whether the respondent has the requested data readily available, what the respondent would do to answer the question).

Contextual factors Geographic location and conditions; political, technological, environmental, and social climate; cultures; economic and historical conditions; language, customs, local norms, and practices; timing; and other factors that may influence the outcomes of interest.¹

Continuous improvement Ongoing learning and evaluation to inform innovation and enhancement of both processes and products over time.

Evaluation A systematic process to determine merit, worth, value, or significance.²

Evaluation question High-level questions used to guide an evaluation.³

Focus group A qualitative research tool in which a small group of people is collectively asked to respond to typically broad, open-ended questions, allowing interaction among the focus group members. These are useful for understanding what issues are important to people and exploring their motivations and behavior patterns.

Global change Climate and other environmental changes that affect ecosystems, people, places, and societies around the world. Climate change includes variations in average temperature, shifts in precipitation patterns, and intensity of storm events.

Impact evaluation Assesses program effectiveness in achieving its ultimate goals.⁴

Logic model Provides a pictorial theory of change showing the process by which change is expected to happen in complex systems, helping evaluators make hypotheses about the connection between users and the NCA.

Network A collection of nodes and the connections between them; for example, a professional organization might maintain a network through which its members (nodes) receive and distribute information (connections).

Network of networks A multilayer network in which the different layers feature different types of nodes, with potential connections between different types of nodes across the layers.

Nongovernmental organization A nonprofit organization that operates independently of any government.

Outcome/effectiveness evaluation Measurement of program effects in the target population by assessing the progress in the outcomes or outcome objectives that the program is intended to achieve.⁵

Participant One who actively contributes in some way to the development or dissemination of the NCA.

Pathway In a logic model, a pathway reflects the causal path by which an action produces change.

Process/implementation evaluation Determines whether program activities are operating or have been implemented as intended.⁶

Program evaluation A systematic method for collecting, analyzing, and using data to examine the effectiveness and efficiency of programs and to contribute to continuous program improvement.⁷

Stakeholder One who has an interest in or concern about climate changes, including researchers, policymakers, decision makers, other diverse groups, and those affected by climate change; in this report, the terms audience and participant are used in place of stakeholder.

Users People, organizations, agencies, and institutions who make use of climate information to take action. Direct users will include those who use the NCA without an intermediary. Indirect users will include those who use materials created from the NCA without necessarily being aware that the NCA is the source.