3

Evolution: SciAct 1.0 to SciAct 2.0

This chapter describes the evolution of the Science Activation program (SciAct) 1.0 in response to recommendations in the National Academies of Sciences, Engineering, and Medicine’s (National Academies’) report NASA’s Science Activation Program: Achievements and Opportunities (NASEM, 2020), noting specific successes and programmatic challenges as well as highlighting potential opportunities for improvement. The chapter then describes the SciAct 2.0 portfolio in depth, including a discussion of SciAct’s new portfolio-level evaluation, and summarizes the committee’s understanding of SciAct 2.0’s successes.

CHANGES IN RESPONSE TO EARLIER NATIONAL ACADEMIES RECOMMENDATIONS

As noted previously, the 2020 National Academies assessment of SciAct found that, although the program had successes during the first five-year period (SciAct 1.0), additional refinements and steps could be taken to better fulfill the program’s overall goals. To this end, the committee outlined a suite of specific recommendations for SciAct 2.0. Within each of the sections that follow, we describe SciAct’s progress in addressing each recommendation from the 2020 National Academies assessment, and we identify opportunities for additional progress, which informed our recommendations for SciAct 3.0.

Refining SciAct’s Focus

2020 Recommendation 1: SciAct should go through a visioning process that brings the portfolio up to date with current research on learning and design, the new federal STEM plan, and evidence-based approaches to broadening participation. This process should also consider how SciAct fits within and contributes to the larger STEM education ecosystem and should provide the foundation for developing actionable and measurable portfolio goals.

2020 Recommendation 2: SciAct should articulate how it expects that the portfolio will leverage NASA assets, how partnerships and networks will be built, and how these actions will lead to desired, measurable outcomes.

Response to Recommendations

In response to these recommendations, SciAct embarked on a visioning process. Activities at the 2019 and 2020 annual meetings involved SciAct projects in this process. Through visioning activities conducted in collaboration with SciAct grantees, SciAct developed specific, concrete program objectives that could be incorporated into individual project objectives. Activities included conversations with principal investigators (PIs) in fall 2019, a workshop to identify and define mid-level objectives (MLOs), and a conversation open to SciAct PIs and project staff to discuss the operationalization of those objectives. This process resulted in a set of MLOs that fall under each of SciAct’s top-level objectives (see Box 3-1).

Project PIs and their project-level evaluators are tasked with identifying the MLOs that are most salient for their projects and including these MLOs in their evaluation plans. The portfolio-level evaluator (described later in this chapter) has engaged in efforts to understand how SciAct projects are incorporating MLOs, and has identified MLOs 1a, 1b, and 3b as the most pursued MLOs. The committee sees the inclusive process for developing the MLOs and their adoption by projects as responsive to 2020 Recommendation 1 and notes many project-level staff told the committee that the MLOs were helpful as they defined their individual project-level goals and focused their programmatic efforts.

Opportunities for Improvement

Further specifying the portfolio’s desired outcomes is a key opportunity. Although both SciAct leadership and portfolio-level evaluators developed distinct program logic models to illustrate the theory of change

underlying the program (see discussion later in this chapter), neither model clearly identifies specific, measurable portfolio-level outcomes or outputs. Therefore, the committee observed an existing opportunity for SciAct to further clarify its vision for how chosen inputs and activities will produce desired results. We discuss how SciAct might approach this opportunity in Chapter 5.

Development of a Coordinated Learning Network

2020 Recommendation 3: SciAct must consider whether the development of a coordinated learning network of awardees across its portfolio is a program priority. If it is a priority, then the program must provide the necessary infrastructure to support a more active network of projects. At the very least, SciAct needs to develop more systematic mechanisms for projects to share best practices and learn from successes and failures.

Response to Recommendation

Following the release of the 2020 National Academies assessment, SciAct leadership identified the learning network concept as a priority in the program’s conceptual framework and took several steps to make this priority salient and to provide the infrastructure to support it. To support a learning network, the program has invested in several areas:

• Creation of additional staff positions to provide support for collaboration across SciAct and with other National Aeronautics and Space

Administration (NASA) programs.

• Establishment of monthly meetings, action groups, and support for building new communities of practice, to support internal sharing and learning within the SciAct network.
• Selection of tools and activities that enable knowledge sharing.
  • In fall 2020, a research team at the Massachusetts Institute of Technology¹ was selected to help SciAct leadership understand the community of SciAct projects, including an analysis of partnerships² leveraged across the portfolio.
  • Action groups, many of which were co-designed at SciAct’s annual meetings, were formalized to support cross-project collaborations around key program priorities, including
    • Broadening participation,
    • Process of (doing) science,
    • Citizen science,
    • Digital learning,
    • James Webb Space Telescope action group,
    • Indigenous community relationships,
    • Women and girls,

___________________

¹ https://science.nasa.gov/sciact-team/mit-media-lab/, Dr. Rachell Connolly (PI).

² https://science.nasa.gov/learn/leverage/

• • • Data visualization,
    • Data fluency/open science, and
    • Diversability.³
  • Some action groups have evolved into communities of practice or affinity groups, each with their own goals and activities, including
    • Holding monthly meetings to share successes and challenges,
    • Providing educational sessions across SciAct projects,
    • Publishing peer-reviewed papers,
    • Sharing resources, and
    • Developing recommendations for SciAct projects and leadership.
• Points of contact (POCs) have been identified within each SMD science division, to serve as internal ambassadors for SciAct. These POCs presented on their division’s current missions and recent advances at SciAct’s 2023 annual meeting.

Based on committee members’ conversations with grantees, efforts to strengthen connections across projects in the portfolio and to enhance opportunities for grantees to learn from each other have been positively received and are useful. These activities, which serve to build out the learning network of grantees, are a major advancement of SciAct 2.0.

Additionally, SciAct issued new solicitations (the Research Opportunities in Space and Earth Sciences [ROSES] 2020 call discussed in Chapter 2) encouraging projects that would address gaps in the portfolio identified in the National Academies 2020 assessment. In response to the recommendation related to developing a coordinated learning network, SciAct STEM Ecosystems⁴ was selected from SMD’s ROSES-2020⁵ grant solicitation to support the development of a network across SciAct projects and to view the portfolio through the lens of a STEM education ecosystem. SciAct STEM Ecosystems convened a meeting in early 2024, which brought together individuals from about 40 SciAct projects. SciAct STEM Ecosystems also conducted an inquiry to explore ecosystem best practices within SciAct. SciAct leadership has been explicit that its programmatic investments are small relative to the investments made in STEM education across the United States. Thus, SciAct looks for leverage points to capitalize on NASA assets—the agency’s content, data, brand, and the excitement of subject matter experts (SMEs).

___________________

³ The term “diversability” is used by SciAct in relationship to supporting diversity among learners.

⁴ https://science.nasa.gov/sciact-team/stem-ecosystems/

⁵ https://science.nasa.gov/researchers/solicitations/roses-2024/research-opportunities-in-space-and-earth-science-roses-2024-to-be-released-february-14-2024/

Opportunities for Improvement

While SciAct projects appear to value the regular check-ins and opportunities to collaborate, some SciAct PIs and project staff expressed the desire for additional opportunities for intentional collaboration. At the same time, some projects indicated limited bandwidth and resources to collaborate. Chapter 4 discusses how SciAct might consider expanding opportunities for projects to learn from one another.

Filling SciAct Portfolio Gaps

2020 Recommendation 4: SciAct should use the opportunity provided by Phase Two to reflect on the current portfolio within the context of the new vision, goals, and logic model. This process should critically review and guide existing projects, be explicit about the rationale and criteria for including new projects, and consider how best to integrate them into the existing portfolio. One important area for consideration is how to ensure that underserved communities receive more focused attention in the next phase of the program.

Response to Recommendation

As part of reviewing the existing portfolio and deciding how to build out the portfolio of projects for SciAct 2.0, SciAct implemented a competitive process to review extension plans submitted by SciAct 1.0 projects. In addition, the ROSES-2020 solicitation (see Chapter 2 for details of this solicitation) was explicit about two specific focus areas: (1) ways that NASA SMEs could engage with the projects, and (2) efforts to broaden participation of underrepresented and underserved learners. A subsequent “gap-filling” ROSES-2021 solicitation was released to cover three focal areas: (1) heliophysics content, (2) dissemination of SMD assets into communities or specific audience networks, and (3) integration of data in science learning. In the committee’s judgment, the SciAct 2.0 portfolio successfully included projects that purposefully involve a diverse set of learners and communities (areas of interest for both ROSES-2020 and ROSES-2021). Eighteen of the 37 non-infrastructure projects in the SciAct 2.0 portfolio focus on specific underserved audiences, including Native American nations in five states, learners on the autism spectrum, people who are blind or have low vision, Black and Latinx communities, and community college students.

The purposeful expansion of the portfolio with careful review of existing projects and use of ROSES competitions to intentionally fill identified

gaps has resulted in a stronger portfolio and is very responsive to 2020 Recommendation 4.

Opportunities for Improvement

Given the strong community formed among existing SciAct projects, embracing new projects often required intentional effort. Bringing additional projects into the SciAct community for SciAct 3.0 will likely also require explicit attention as new personnel and partners become involved.

Deepening Commitment to Broadening Participation

2020 Recommendation 5: SciAct should deepen its commitment to broadening participation by using evaluation measures that go beyond counting numbers of individuals who represent specific groups. In order to do this, SciAct must identify ways that the portfolio as a whole could draw upon and implement evidence-based strategies for broadening participation.

Response to Recommendation

The ROSES-2020 solicitation’s emphasis on broadening participation resulted in several proposals that articulated plans to work deeply and strategically with groups that are traditionally underrepresented and underserved in STEM. Through a peer-review process, SciAct selected a number of these projects for inclusion in SciAct 2.0, and, as noted in the previous section, the committee views this expansion of the audiences served by projects within the portfolio as a positive step forward.

SciAct is also working to provide support for projects that are engaging with groups that have been traditionally underserved and underrepresented in STEM. A monthly learning community, one of the activities of the Broadening Participation action group, supports sharing of diversity, equity, inclusion, and accessibility (DEIA) expertise across projects by featuring a presentation in each virtual meeting, during which a project team shares insights into their successful DEIA efforts.⁶

Project-level evaluations of projects working in partnership with communities include documentation and examination of how projects are involving and engaging communities. In some projects, these efforts include capturing how SciAct teams incorporate traditional, Indigenous, and ecological knowledge. Also, some formative evaluations are capturing how

___________________

⁶ This sentence was changed after release of the report to correct an error regarding the origin of the Broadening Participation action group.

projects are engaging in a co-design process with community members and are feeding that information back to project PIs to help improve their efforts, as well as sharing learnings with the broader SciAct community. This knowledge sharing has occurred at large meetings, and it is also described in the 2022 NASA Science Activation Impact Report; however, it is unclear the extent to which this is documented in proposal narratives or project annual reports, as well as how this has been actualized and measured within and across projects.

SciAct projects also engage in internal discussions around DEIA to further share learnings from work with specific demographic groups. In addition to informal sharing of challenges and successes, DEIA-focused sessions were convened at both the annual meeting and education ecosystem meetings.

SciAct has recognized the need to deepen the knowledge of grantees and SciAct staff around supporting DEIA and working with communities. Portfolio expansion offers an opportunity for SciAct to continue to deepen this work. The efforts of project-level evaluators to document partnerships and co-design processes could inform portfolio-level evaluations. These developments are all responsive to 2020 Recommendation 5.

Opportunities for Improvement

An opportunity exists for SciAct to be more intentional, across its portfolio, to include a balance of both learner- and community-centered approaches, as well as to more deeply support community-centered approaches already operating within projects. Chapter 4 discusses potential opportunities for NASA to further expand and deepen its DEIA work beyond its current “broadening participation” approach. Specific to the evaluation component of 2020 Recommendation 5, which called for SciAct to use measures that go beyond “counting numbers of individuals who represent specific groups,” Chapter 4 describes how SciAct might more comprehensively represent its successes and challenges by engaging with more narrative, qualitative approaches to evaluation.

Dialogue with Missions and Scientists

2020 Recommendation 6: SciAct should build ongoing opportunities for dialogue with NASA Science Mission Directorate’s missions and scientists.

Response to Recommendation

As part of implementing SciAct 2.0, in an effort to strengthen connections to SMD’s missions and scientists, SMD science division leads were

identified as POCs, to facilitate internal NASA communication around SciAct. POCs present their current missions at the annual SciAct meeting, keeping the SciAct community appraised of ongoing SMD activities and assets. SciAct projects also engage with POCs to establish connections with SMD divisions.

Several projects are working explicitly to make connections between SciAct and the larger professional scientific community, both within NASA and beyond. Two of these projects, namely Planetary Resources and Content Heroes⁷ and NASA Community of Practice for Education (SCoPE),⁸ were funded through ROSES-2020. Further progress on this recommendation is demonstrated by NASA’s Universe of Learning⁹ project’s work with the astrophysics community.

Establishing POCs, involving them in SciAct annual meetings, and supporting projects designed to connect with the space science community are all positive steps forward. Thus, in the committee’s view, SciAct has been responsive to 2020 Recommendation 6.

Opportunities for Improvement

In conversations with SciAct project PIs, the committee heard that finding and connecting with SMD SMEs often relied on PIs’ existing individual connections, so some projects were more connected than others. Opportunities to continue evolving the ways SciAct staff and SMD POCs connect SciAct project teams with SMD mission teams exist both within NASA and across the scientific community. SciAct could connect more closely with other NASA programs, such as NASA Engages¹⁰ in the Office of STEM Engagement, which is composed of NASA experts who share NASA missions and content at educational, professional, civic, and other public venues.

Expertise and Evaluation

2020 Recommendation 7: SciAct should create an independent mechanism to obtain ongoing, real-time advice from individuals with expertise in learning and design, the larger policy context of STEM education, partnering with local communities, broadening participation in STEM, and science content relevant to the missions of NASA’s Science Mission Directorate. Among other responsibilities, these experts should inform the new visioning and planning process.

___________________

⁷ https://science.nasa.gov/sciact-team/planetary-resources-and-content-heroes/

⁸ https://science.nasa.gov/sciact-team/smd-community-of-practice-for-education/

⁹ https://universe-of-learning.org/home

¹⁰ https://my.nasa.gov/engages/s/

2020 Recommendation 7a: With input from these experts, SciAct should consider whether and how a portfolio-level evaluation could strengthen the focus of the program and ensure that projects in the portfolio are effectively meeting overarching SciAct program goals and objectives.

Response to Recommendations

Following the 2020 National Academies assessment, SciAct leadership invited individuals with scholarly expertise relevant to a suite of content area concerns highlighted in the 2020 report (e.g., evidence-based approaches to STEM education, broadening participation, engaging with neurodiverse learners) to present at SciAct annual and monthly meetings, and SciAct has used other National Academies reports focused on STEM education to guide its work (NASEM, 2016, 2018b, 2022). Authors of the 2020 National Academies assessment and National Academies staff members engaged in the visioning process and helped to develop MLOs. However, there does not appear to be a mechanism for regular infusion of the expertise suggested in 2020 Recommendation 7.

In response to 2020 Recommendation 7a, SciAct carried out a competitive process to engage a portfolio-level evaluator and selected Pacific Research and Evaluation, LLC (PRE) in December 2020. PRE’s activities have included developing a portfolio logic model, exploring the use of MLOs across projects, and working to identify shared measures for evaluating programmatic success across SciAct projects. PRE’s team has engaged with the independent project evaluators associated with each SciAct project, whose primary role is assessing the project’s progress and providing feedback to both SciAct and project leadership to inform ongoing improvements. A working group of project-level evaluators has been established to facilitate sharing of knowledge and best practices between projects, providing another source of information that may inform overarching SciAct program goals and objectives.

Opportunities for Improvement

With respect to 2020 Recommendation 7, opportunities remain for SciAct to develop a mechanism for regularly obtaining outside expertise to supplement the SciAct community’s robust internal networks. With respect to 2020 Recommendation 7a, a key opportunity for SciAct moving forward is articulating which measures of programmatic success are important to capture in the portfolio-level evaluation (see discussion of the portfolio-level evaluation below). Opportunities also exist with consideration to the

portfolio-level data-collection process, including which questions are asked and how, which data-collection methods are employed, and the differential capacities of expected data collectors (i.e., project teams/evaluators, SciAct staff, PRE).

THE CURRENT PORTFOLIO: SCIACT 2.0

The following sections provide a brief description of the current portfolio (SciAct 2.0), shaped by SciAct’s response to the 2020 National Academies assessment, and describe SciAct 2.0’s approach to measuring success. This portfolio description is based on annual SciAct Impact Reports from 2022 to 2023 and the 2021 PRE portfolio-level evaluation plan (PRE, 2021).

Describing SciAct 2.0

Annual SciAct Impact Reports provide both a portfolio-level characterization of impacts and accomplishments along with project-level summaries of project goals, audiences, educational settings, delivery models, accomplishments, key partners, and outputs (i.e., numbers of activities, participants, reach, etc.). These reports show that all five SMD divisions are represented in the SciAct 2.0 portfolio (2022/2023):

• Earth science: 11/10 projects
• Planetary science: 3/3 projects
• Heliophysics: 4/4 projects
• Astrophysics: 3/3 projects
• Biological and physical sciences: 1/1 project
• Division cross-cutting: 15/16 projects

Thirty-five of the 37 non-infrastructure SciAct 2.0 projects work with educators and/or students in formal and/or informal educational settings; only the SMD SCoPE and STEM Ecosystems¹¹ projects engage solely with NASA SMEs or SciAct project team members, respectively. Over 60% of SciAct projects work with formal and/or informal educators, and most projects either engage directly with or develop resources for youth ages 5–17. Two projects specifically engage with undergraduate students or their instructors: Eclipse Ambassadors Off the Path,¹² which trained amateur astronomers and undergraduate students to provide community outreach

___________________

¹¹ https://science.nasa.gov/sciact-team/stem-ecosystems/

¹² https://science.nasa.gov/learning-resources/science-activation/eclipse-ambassadors-off-the-path-reaching-underrepresented-audiences/

during the 2023 and 2024 eclipse events, and NASA Community College Network,¹³ which provides community college instructors with resources, a community of practice, and connections with NASA SMEs. Among the 30 projects working in informal education or out-of-school settings, approximately 35% provide citizen science opportunities, 30% share digital learning opportunities, nearly 50% work with science centers/museums, and 30% work with libraries. Projects also involve historically underserved communities: 8 projects include people with differing abilities, 2 projects work with neurodiverse audiences, 25 projects work with communities with diverse racial/ethnic identities, and 18 projects engage with rural communities (PRE, NASA SciAct Portfolio Groupings Public tool, as provided to the committee).

SciAct 2.0’s theory of action is a “collective impact, network-of-networks approach” (NASA, 2023a, p. 4). NASA resources and expertise are the inputs in this approach; SciAct projects implement a variety of activities (e.g., workshops, curricula, online educational resources and tools, engagement activities and public events, citizen science programs, communities of practice, learning ecosystems) to engage various audiences to meet a selection of MLOs. Outputs are mostly dissemination oriented, captured through measures of reach (e.g., number of learner interactions by geographic location [NASA, 2023a, p. 5]), collaboration (e.g., number of partnerships with external organizations and/or with other SciAct teams [NASA, 2023a, pp. 6–7]), participation of NASA SMEs (e.g., number of SMEs participating in SciAct activities by geographic location [NASA, 2023a, p. 8]), peer-reviewed publications (NASA, 2023a, p. 9), availability of learning resources (e.g., number of learning resources by NASA science divisions [NASA, 2023a, p. 9]), and broadening participation (e.g., number of projects within specific communities and number of DEIA and belonging topics shared with the SciAct community [NASA, 2023a, p. 10]). The collective impact approach posits that implementation of these SciAct project activities will contribute to meeting the four top-level objectives of SciAct described in Chapter 2 (Figure 3-1).

Dimensions of SciAct 2.0 Portfolio

SciAct 2.0 uses four descriptive categories to characterize and organize its projects (NASA, 2023a):

1. SMD divisions: Astrophysics, biological/physical sciences, Earth science, heliophysics, and planetary science.
2. Audience age: Ranging from 0 years old and up.

___________________

¹³ https://science.nasa.gov/sciact-team/nasa-community-college-network/

3. Education settings: Informal/out of school, formal education, citizen science, and professional.
4. Delivery models: Independent/self-directed; facilitated learning, guided by informal educators; delivered by formal educators; peer-professional learning.

In addition to these categories, the committee observed that SciAct 2.0 projects could be characterized in two ways that are not currently documented in the 2022 and 2023 Impact Reports: type of engagement and nature of the engagement strategy.

Type of Engagement

SciAct 2.0 contains projects that range in depth of engagement—that is, the length of time participants engage and how deeply they are involved with the scientific ideas and practices. For instance, NASA’s Heliophysics Education Activation Team (HEAT)¹⁴ eclipse event, held in Dallas in April 2024, engaged a large number of people in a short-duration interaction. Other activities, like Learning Ecosystems Northeast¹⁵ or the NASA Community College Network, focus on long-term, in-depth interactions with small numbers of people. A spectrum of approaches contributes to a well-functioning science learning ecosystem that supports all learners (Ostman, 2024). For instance, a large science event can be a “spark,” generating interest or resources that can be used in many contexts, while long-term events involving the same group of learners allow more time to explore scientific methods and processes. Large-scale dissemination can reach many people who are already inclined toward science, but focused engagement can provide more welcoming environments for nurturing science identity, supporting participants’ agency as learners and users of science, and exploring intersectionality of science and culture (Kim & Sinatra, 2018). Research shows that learners benefit from opportunities to connect informal and formal STEM experiences, either through their own choices or via curated pathways, suggesting that the impact of an approach may be multiplied through its connections to other approaches (Hussim et al., 2024). Furthermore, knowledge and products resulting from one approach can inform other approaches—for instance, a curriculum developed with a small group of learners may subsequently be used with a larger group of learners—which is an additional benefit of communication between projects with diverse approaches.

___________________

¹⁴ https://science.nasa.gov/sciact-team/nasa-heliophysics-education-activation-team/

¹⁵ https://science.nasa.gov/sciact-team/gmri/

Nature of the Engagement Strategy

As discussed in Chapter 2, scientific research in the United States is increasingly leveraging community-oriented approaches. Similarly, SciAct 2.0 projects also span a spectrum of engagement strategies, ranging from projects that are primarily science centered to those that are deeply community centered. Efforts that focus on disseminating science or the “breaking news” of Earth and planetary discovery are on the science-centered end of the spectrum (e.g., eclipse-focused projects, science festivals, opportunities to meet NASA personnel, and educational programs and experiences devoted to missions or topics, like heliophysics). While designers of programs that utlize these approaches often employ knowledge about participants’ backgrounds or interests and program activities can be interactive (e.g., a question-and-answer session between participants and astronomers at an eclipse festival), they center scientific content, resources, expertise, and experts. Such approaches can be quite effective for meeting their stated goals: research on festivals, for example, revealed that content-driven approaches can increase participants’ interest in science (Illingworth et al., 2015; Jensen & Buckley, 2014) and are particularly attractive if they involve direct interaction with scientists (Boyette & Ramsey, 2019). Dissemination-driven approaches can also increase awareness of science careers (Canovan, 2019; Munn et al., 2018). Moreover, these approaches may amplify the resources present in informal learning institutions, contributing to the overall science ecosystem (Bevc et al., 2016). A weakness of dissemination-driven approaches, however, is that they may primarily attract participants who are more affluent and educated than the general population (Falk & Needham, 2011; Kato-Nitta et al., 2018; Kennedy et al., 2018; Martin, 2017; Nielsen et al., 2019).

Another class of SciAct 2.0 projects focuses on educational opportunities associated with inviting non-scientists to participate in the processes of scientific research, or educational citizen science.¹⁶ SciAct’s educational citizen science projects intentionally support participants’ learning outcomes as well as participants’ scientific contributions, through activities like data collection or image classification. In contrast, other NASA citizen science programs primarily use citizen science as a tool for advancing scientific research, with little focus on participants’ learning.

___________________

¹⁶ The term “citizen science” is problematic, especially in the United States. “Citizen” was originally intended to describe a participant as a “citizen of the world” (Bonney, 2021), but the word may not be welcoming to those living in the United States without documentation (Cooper et al., 2021). “Community science” is sometimes used as a replacement, but in this report, we reserve “community science” for projects that center community knowledge (including knowledge beyond the sciences) and in which community members collaborate to set project priorities, goals, outcomes, and measures.

Educational citizen science projects can afford positive learning outcomes across all strands of individual science literacy (Finger et al., 2023; NASEM, 2018b). Outcomes of educational citizen science projects can be improved by attending to the interests and skills of non-scientist participants (NASEM, 2018b). Educational citizen science projects can provide access to authentic scientific practices that help people understand the intrinsic uncertainty in all scientific work, including that of NASA (Chen et al., 2019; Fishoff & Davis, 2014; Star, 1985). While SciAct can disseminate established scientific information, providing access to authentic scientific practices that manage intrinsic uncertainty is a major opportunity for which educational citizen science is ideally suited.

A similar class of participatory SciAct projects leverages NASA data, people, and assets in ways that are highly relevant in local contexts and represent “science-as-a-service.” For example, in the Sea Level Education, Awareness, and Literacy (SEAL)¹⁷ project, NASA SMDs work directly with SMEs who present NASA’s satellite-based sea-level data in ways that address local issues and make those data available to local decision makers. In such projects, community priorities drive the overall agenda, which implies a respect for community knowledge. NASA SMEs provide scientific expertise, but community expertise defines pressing needs or issues, such that scientific data, assets, and people can be leveraged in ways that have tangible community impact. As demonstrated by SEAL, science that serves society may be particularly relevant in the face of large-scale community threats like climate change.

In these two kinds of projects—science-as-a-service and educational citizen science—community expertise and leadership play complementary roles. In citizen science, goals are set by scientists, but the interests, skills, and local expertise of community members are critical; in the case of science-as-a-service, scientific expertise is critical, but community goals determine the uses of that expertise.

Finally, some SciAct 2.0 projects intentionally combine community leadership and expertise in both their activities and goal setting, which we will refer to as community approaches to science. In this mode of interaction, science is positioned alongside other types of knowledge, and scientific experts work with community members who bring their own expertise. These efforts often focus on community priorities that include science but extend to issues beyond science. In honoring heterogeneous knowledge systems, such projects often work with cultural communities that have been underserved or even exploited by Western science. For example, Native Earth | Native Sky¹⁸ co-creates culturally relevant Earth

___________________

¹⁷ https://science.nasa.gov/sciact-team/seal/

¹⁸ https://science.nasa.gov/sciact-team/native-earth-native-sky/

and sky STEM curricula with three Native American nations in Oklahoma, by interweaving stories, language, and culture into formal classroom Earth and space curricula.

A relationship exists between the various modes of community engagement and the breadth and depth of engagement. Smaller projects tend to be more amenable to community or co-created science or science education, while larger projects necessarily tend to focus on science-centered dissemination strategies. As discussed earlier, these approaches can be connected—a small, focused, in-depth community-science approach could result in curricula well suited for widespread dissemination within relevant communities.

Results from Portfolio-Level Evaluation

NASA SMD selected PRE as the portfolio-level evaluator to help SciAct identify the impact of its portfolio as a collective, and to describe the portfolio’s progress in accomplishing its stated objectives (PRE, 2021). As part of this work, PRE developed the following:

• A suite of tools intended to help the SciAct community build a shared understanding of SciAct 2.0 and its component projects, including
  • An MLO Summary Dashboard illustrating the distribution of projects pursuing individual MLOs and project activities designed to meet each MLO;
  • A Measurement Tools Landscape, showing methods projects are using to assess their efforts to address MLOs; and
  • An Audience Dashboard, showing the audience groups with which each project engages.
• A logic model for portfolio evaluation (PRE, 2021) (Figure 3-2).
• A suite of common measures to be used across SciAct projects, as appropriate, to understand collective impacts. The recommended common measure, agreed upon at the SciAct annual PI meeting in November 2023, was a single-item measure for assessing STEM identity, called STEM Professional Identity Overlap (STEM PIO-1) (McDonald et al., 2019). This metric is connected to MLO 1a (see Box 3-1); PRE notes that 22 projects in SciAct 2.0 are pursuing MLO 1a as one area of focus.

During 2023, PRE held a series of shared learning sessions with SciAct team members to better understand how MLOs were defined and operationalized across projects. Takeaways from these sessions included the following:

• Most team members viewed MLO 1b (Provide opportunities for participants to engage with the disciplinary content related to NASA science and engineering) and MLO 1c (Increase number of and frequency with which NASA SMD assets are used by learners across the United States) as project outputs versus outcomes, with a focus on quantifying people, events, or materials or the number of participant engagements with content and/or SMD assets.
• Project staff wanted clarity from SciAct leadership on most MLOs, especially relative to expectations for project reporting and definitions of terms (e.g., “learner,” “engagement,” and “NASA SMD asset”).
• Project staff interpreted MLO 2a (Advance participants’ understanding of science and its processes using NASA SMD assets) as focused on learning the process of science, particularly how science is done at NASA.
• Project staff interpreted MLO 3a (Increase participation in learner-centered experiences based on NASA SMD assets) as both an output (e.g., quantifying implementation and engagement by tracking student participation in activities) and, less frequently, as an outcome in which the impact of an activity on participants was assessed. Staff generally expressed that this MLO was easy to understand.
• Project staff appreciated that MLOs were intentionally broad, so as to be applicable to all projects, but staff were unclear on how to report their progress toward MLOs.

DEFINING SUCCESS FOR SCIACT 2.0

SciAct seeks to leverage NASA science content and SMEs to further the understanding and pursuit of science among individuals of all ages. The committee identified several characteristics that demonstrate programmatic “success” for the work accomplished in SciAct 2.0. Although this list is not exhaustive, it represents a suite of achievements that the committee feels signal SciAct 2.0’s ongoing progress toward meeting its top-level objectives (see Chapter 2):

• Connecting individuals with science through SciAct project activities.
• Creating a diverse portfolio of projects that
  • Are designed for NASA outreach and intentional participant engagement;
  • Represent the full spectrum of disciplines embodied by SMD’s divisions;
  • Include a wide range of learners and activities;

• • Focus on broadening participation of groups historically underrepresented in SMD STEM fields; and
  • Reach learners across U.S. states and territories and beyond.
• Leveraging NASA assets including SMD experts, science discoveries, and SMD resources.
• Establishing successful partnerships, including those that strike a balance between dissemination and community-engagement modes, as described above.
• Using evidence-based and best-practice approaches that support learning across the entire portfolio, emphasizing learner- and community-centered instructional design and practices.

SciAct 2.0 has made observable progress toward meeting its stated objectives: that is, in the wake of the 2020 National Academies assessment, considerable efforts by SciAct leadership and PIs resulted in tangible improvements in the areas listed above. Additionally, these successes can guide potential pathways for a future SciAct 3.0—areas in which SciAct has made changes may serve as instructive examples illustrating how change in the SciAct portfolio happens and where attention is best focused.

The following conclusions summarize the progress of SciAct 2.0.

Conclusion 1: The Science Activation program has made significant progress in response to the recommendations put forth in the 2020 National Academies assessment. Notable achievements include broadening the kinds of communities engaged and creating and sustaining a collaborative network of projects.

Conclusion 2: The creation of mid-level objectives (MLOs) was an important step in clarifying and refining the Science Activation program’s (SciAct’s) goals. Developing MLOs involved collaboration across the entire SciAct portfolio, allowing for input from multiple principal investigators and project staff. While this is a step forward, further refinement of MLOs could help clarify how individual projects could best use/work toward these objectives.

Conclusion 3: The portfolio-wide evaluation, while helpful in many ways to the Science Activation program’s administration, favors single metrics (e.g., a survey of science identity) over documenting and describing the successes and challenges of specific awards to capture overall impact and inform progress for the entire portfolio.

Conclusion 4: The current logic model developed by Pacific Research and Evaluation does not clearly identify specific outcomes or outputs,

nor does it specify how program and project inputs and activities are expected to lead to desired outcomes.

Conclusion 5: Independent project-level evaluators are an important resource for individual projects and for the overall portfolio. Tailoring evaluations to individual projects captures projects’ unique contributions and can inform future work. It is unclear how individual project evaluations inform the portfolio-wide evaluation.

Conclusion 6: The Science Activation program has successfully built a portfolio of projects that are diverse with respect to geographic reach, scientific discipline, types of audiences, and types of partnerships leveraged. The committee noted two additional dimensions represented across projects: deep engagement with a small audience versus lesser-depth engagement with a larger audience, and community-centered approaches versus science-centered approaches.

This page intentionally left blank.