5

Recommendations for SciAct 3.0

In the wake of this second assessment of National Aeronautics and Space Administration’s (NASA’s) Science Activation program (SciAct), the National Academies of Sciences, Engineering, and Medicine’s (National Academies’) Committee to Assess NASA Science Activation 2.0 has identified a suite of recommendations that, if implemented, can further propel SciAct toward meeting its goals. SciAct has made considerable progress since the first National Academies assessment (NASEM, 2020): not only has SciAct acted upon every recommendation outlined in the first report, but the program has also taken additional steps to support the projects in its portfolio. Building on these accomplishments, this chapter reiterates opportunities for improvement observed across the current assessment. These observations are intended as probes for SciAct as it plans its 3.0 programming: the committee offers these ideas in the spirit of helping SciAct sharpen its focus and expand upon its initial efforts. Formal recommendations, intended to capitalize on SciAct’s ongoing work, are provided at the end of the chapter.

OPPORTUNITIES FOR IMPROVEMENT: INTENTIONAL COMMUNICATION

This report identified several areas in which SciAct leadership could strengthen the program through intentional decision making, clearly communicated to the entire SciAct portfolio. Various examples illustrate instances in which intentional decisions could support programmatic success. First, although the 2020 National Academies assessment delineated

an operational theory of change, SciAct principal investigators (PIs) and project staff repeatedly identified the need for further guidance outlining how leadership’s expectations of SciAct projects will serve the portfolio’s objectives. As discussed in Chapter 2, the logic model developed by SciAct’s portfolio-level evaluators (Pacific Research and Evaluation [PRE]) does not clearly identify specific outcomes or outputs, nor does it reflect how SciAct expects its program and project inputs and activities to lead to desired outcomes.

The committee noted additional decision-making opportunities for SciAct leadership, through which decisions could inform the actions and directions of others—that is, by making an intentional decision in each of these realms, other actors in the portfolio will be enabled to pursue their own programmatic work more effectively. We offer the following provocations:

• What types of equity-related goals or outcomes does SciAct hope to effect?
• What balance does SciAct wish to strike across dissemination-driven and community-centered approaches to programming?
• What does “success” look like in terms of connecting with Science Mission Directorate (SMD) assets?
• What types of portfolio-level success would SciAct like to demonstrate/communicate to outside stakeholders?

COMMUNICATING SCIENCE ACTIVATION IMPACTS AND RELEVANCE ACROSS NASA AND BEYOND

The SciAct portfolio clearly contributes to NASA’s strategic goals and objectives, particularly Strategic Objective 4.3: build the next generation of explorers (NASA, 2022). Sharing data, information, and knowledge with the public is very much part of NASA’s vision and responsibility as a federal agency. SciAct is analogous to other programs within SMD, as it has its own goals, objectives, and program-specific requirements. There is an opportunity for SciAct to explicitly connect its objectives, successes, and impacts to NASA’s strategic priorities and decadal plans in ways that inform NASA, particularly on community-centered practices. The decadal plans released to date indicate significant challenges in building and sustaining a diverse pipeline of professionals. SciAct could help NASA engage with communities historically underserved by and underrepresented in science, thereby building interest in and excitement about STEM careers within those communities.

SciAct has clearly made efforts to share its learnings beyond NASA, including with the broader education community. Several projects have published their experiences and learnings in education research outlets,

and presentations highlighting SciAct projects are common at professional society annual meetings. Given the unique design of SciAct within the federal landscape, there is an opportunity to learn from the program’s ongoing work as a professional learning community: What has SciAct learned that could benefit other programs? Where and how has SciAct succeeded in supporting learning across the portfolio? Documenting these observations could benefit the larger education landscape, and SciAct leadership may want to consider how to best communicate these successes to NASA and beyond.

FINAL THOUGHTS AND RECOMMENDATIONS

The committee commends SciAct leadership for the progress achieved over the past five years in addressing the recommendations put forth in the 2020 National Academies assessment. The SciAct portfolio clearly contributes to NASA’s strategic goals and objectives in meaningful ways. SciAct has the opportunity to explicitly connect its goals, objectives, and successes/impacts to NASA’s strategic priorities and decadal plans in ways that inform NASA, particularly on community-centered practices. The unique design of SciAct within the federal landscape provides an opportunity to learn from SciAct’s ongoing work as a professional learning community, and for sharing its successes within NASA and beyond. Designated SciAct staff could lead the coordination of such efforts.

To complement SciAct’s decision making in the above areas, we offer several formal recommendations for immediate action. We believe that these recommendations will bring SciAct even closer to meeting its stated objectives.

Recommendation 1: The Science Activation program (SciAct) should be transparent about the goals of its portfolio evaluation and clear about its intentions for using evaluation results. Furthermore, in light of the portfolio’s diverse nature, the evaluation should consider measuring SciAct’s impact through multiple measures instead of a single measure. To capture the breadth and depth of project work, the portfolio-level evaluation should utilize methods that highlight stories and narratives of learning and change.

As articulated in Chapter 3, the decision to hire a portfolio-level evaluator is a direct response to a recommendation outlined in the 2020 National Academies assessment. PRE made efforts to provide portfolio-level feedback, but the current committee recognizes that SciAct’s diverse portfolio may not be well served by the use of single, quantitative measures of programmatic success. The 2020 National Academies assessment outlined a series of potential goals for a portfolio-level evaluation, including

• Deepening understanding of how individual projects align with SciAct goals;
• Improving portfolio-level coherence throughout all stages of project planning, implementation, and assessment;
• Surfacing unique and promising trends within the portfolio;
• Identifying areas for improvement on a continual basis;
• Representing one mechanism for feedback between SciAct network members and SciAct leadership;
• Developing potential models for understanding SciAct’s impact; and
• Directly facilitating participation routines across the network that support participatory knowledge building, problem and goal identification, and impact assessment approaches (NASEM, 2020, p. 86).

While the PRE evaluation considered many of these potential goals, SciAct could benefit from clarity and transparency around the specific goals the portfolio-level evaluation is pursuing as well as how SciAct intends to use the evaluation’s findings. It is important for SciAct project staff and evaluators to understand how they are expected to respond to the portfolio-level evaluation, as well as to feel confident that their successes and challenges will be reflected in evaluation efforts. To achieve this goal, the portfolio-level evaluation may consider expanding its scope to capture the portfolio’s diversity of approaches to supporting STEM learning and engagement and aggregate project contributions across a range of axes rather than a single measure.

Recommendation 2: In building the next iteration of its portfolio, the Science Activation program (SciAct) should consider further investment in community-centered approaches to programming. In expanding this investment, SciAct should ensure that future funded projects attend to modes of engagement and learning that center community-engaged approaches.

The committee was impressed by SciAct’s success in deepening its commitment to broadening participation in STEM through the addition of a suite of new projects in SciAct 2.0. Since the 2020 National Academies assessment, the value of approaching educational opportunities from a community-centered lens has been increasingly appreciated throughout the education research landscape. As described in Chapter 4, community-centered approaches to STEM engagement overlap considerably with SciAct’s programmatic objectives. Thus, it is important for SciAct to intentionally seek out new projects with community-centered approaches, with the overall goal of expanding the portfolio’s capacity in this area.

Recommendation 3: As the Science Activation program continues its commitment to broadening participation, it should develop strategies for engaging with audiences and communities that have been historically excluded from NASA and science generally, and place priority on (1) supporting existing projects that include those groups and (2) expanding the number of projects that work with those populations.

SciAct has made significant progress in expanding the diversity of groups engaged in its work. These efforts could be further strengthened by intentionally empowering and collaborating with audiences and communities that have typically been excluded by traditional scientific endeavors. To engage in this work, it will be important for SciAct to pay careful attention to how projects are engaging with their audiences: it is not enough to merely invite audiences into traditional science experiences—supporting participants’ identities in science encompasses rethinking the involvement of audiences in project work from the outset. For example, whose questions are answered through participation in a SciAct project? Who dictates the pace of the work? For projects in the current portfolio, how can existing relationships be augmented to deepen participants’ experiences? By developing a suite of programmatic strategies aligned to SciAct’s equity goals, SciAct could be better positioned to meet its primary objectives.

Recommendation 4: The Science Activation program should refine its logic model to reflect the theoretical underpinnings and theories of change by which program and project inputs and activities are expected to lead to desired outcomes, particularly with respect to community-centered approaches in which communities may set priorities and outcomes.

SciAct leadership has an opportunity to more clearly articulate its logic model so that all actors in the portfolio understand exactly how their work is intended to lead to desired outcomes. Specific refining can reflect the ways in which SciAct’s expanding commitment to community-centered approaches is intended to serve the portfolio’s goals, particularly when communities may see conflict between their own goals and NASA’s goals. It is important for a SciAct logic model to carefully capture programmatic inputs, actions, outputs, and outcomes such that all projects can see the relationship between meeting their own goals and meeting the portfolio’s mid-level and top-level objectives. In expressly articulating this logic model, SciAct can attend to the richness afforded by the range and variety of portfolio approaches to supporting learning.

Recommendation 5: Given the importance of leveraging NASA assets for meeting the goals of the Science Activation program (SciAct), SciAct should clarify its definition of subject matter expertise. Specifically, while all projects require subject matter expertise, only some projects will leverage NASA subject matter experts (SMEs) as an identified NASA asset. SciAct should specify what is meant by SMEs and use that definition when building a portfolio that balances multiple types of asset use.

As discussed in Chapter 4, considerable confusion was noted across the portfolio around the definition of the term “subject matter expert” (SME). Given the multiple types of expertise necessary for the diversity of SciAct projects, it is important for SciAct to clarify the kinds of expertise projects require in their implementation. NASA SMEs will be an appropriate NASA asset for some projects but not all. It is important for projects to understand what “counts” as a NASA SME, so they can identify whether the expert they are using is considered a NASA asset. The SciAct portfolio will naturally make use of a range of NASA assets across projects.

Recommendation 6: The Science Activation program (SciAct) should continue to build on the network structures that have been created, and develop those structures in a professional learning community, such that separate project teams can better generate and share knowledge across projects in the SciAct portfolio.

The committee was impressed by SciAct’s commitment to building out a cross-project network structure, and we suggest that SciAct continue that investment. As noted in Chapter 5, SciAct can pursue multiple strategies toward even deeper success in this area, including more coordinated learning opportunities at annual and monthly PI meetings, devoted staff time for supporting network priorities, and new mechanisms for knowledge sharing. Such forms of investment could leverage the portfolio’s considerable strengths toward supporting even stronger project implementation.

As part of building on these network structures, several important questions emerge for SciAct’s consideration. First, given the limited capacities of SciAct PIs and project staff, the committee recognizes the importance of incentivizing participation in professional learning communities so that individual investment of time is both valued and valuable. Furthermore, given that SciAct is committed to a vision of STEM learning that reflects the most current research and practice in the field, we recognize the importance of onboarding new practitioners into these contemporary frameworks and approaches. Future iterations of SciAct may benefit from intentional opportunities to support initial baseline learning for new PIs and project staff.

This report represents the committee’s attempt to harness SciAct’s considerable efforts to improve the performance of its portfolio. Based on our understanding of the evidence, we believe that with additional intentional decision making around operational strategy, SciAct can more effectively pursue its top- and mid-level objectives. We remain deeply optimistic about SciAct’s role in supporting STEM engagement for learners across the country, and we are honored to support SciAct as it continues to meet the nation’s STEM education needs.

This page intentionally left blank.