Summary
BACKGROUND AND TASK
Since 1959, the National Institute of Standards and Technology (NIST) has engaged the National Academies of Sciences, Engineering, and Medicine annually to assemble expert panels consisting of professionals from academia, industry, and other scientific and engineering communities and assess the quality, effectiveness, and resource adequacy of NIST’s six measurement and standards laboratories.1 NIST initiates these evaluations through a contract with the National Academies. For fiscal year 2024, NIST tasked the National Academies with evaluating its Engineering Laboratory (EL). This review by the Panel on Assessment of the National Institute of Standards and Technology (NIST) Engineering Laboratory is based on read-ahead materials, a site visit where the panel toured the laboratory and engaged with EL researchers in one-on-one discussions, a session with postdocs and junior staff, and post-meeting questions. Drawing on its collective expertise, the panel reviewed EL based on the statement of task and provided relevant recommendations.
The statement of task has four key components. First, the panel was tasked to evaluate EL’s technical programs, comparing the quality of its research to similar international programs, and determining whether the programs are sufficient for EL to meet its objectives. Second, the panel was tasked to assess EL’s portfolio of scientific and technical expertise, evaluating whether it is world-class and how effectively it supports EL’s programs and objectives. Third, the panel was tasked to review the adequacy of EL’s facilities, equipment, and human resources in supporting its technical work and overall goals. Finally, the panel was tasked to evaluate EL’s effectiveness in disseminating its program outputs, including how well these efforts are driven by stakeholder needs, the comprehensiveness of the dissemination and technology transfer mechanisms, and how effectively EL monitors stakeholder use and the impact of its output.
ENGINEERING LABORATORY
EL comprises five divisions, including the Building Energy and Environment Division, Fire Research Division, Intelligent Systems Division (ISD), Materials and Structural Systems, and Systems Integration Division. These divisions are housed on the NIST campus in Gaithersburg, Maryland. These divisions contribute to the EL mission to promote “U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology for engineered systems in ways that enhance economic security and improve quality of life” (NIST 2021).
___________________
1 The laboratories are the Center for Neutron Research, Communications Technology Laboratory, Engineering Laboratory, Information Technology Laboratory, Material Measurement Laboratory, and Physical Measurement Laboratory.
KEY RECOMMENDATIONS
What follows are overarching themes and related key recommendations for EL as a whole. They represent things that all of the subpanels observed in the course of the meeting. There are also chapter-specific recommendations in each chapter. Chapter 7 presents the overarching themes, key recommendations, and chapter-specific recommendations in one place.
Strategic Direction and Strategic Planning
The panel found that the strategies and strategic directions guiding EL’s work were not clearly defined. To properly assess the effectiveness of EL’s programs, the panel needed a clearer understanding of EL’s goals and objectives. How does EL ensure that its research areas align with the mission of the laboratory, NIST as a whole, and with stakeholder needs? The panel was not provided with a strategic plan, short- or long-term implementation plans, key performance indicators, or reporting requirements that could help to answer these questions.
Without clear strategic goals, evaluating whether EL has the necessary expertise, budget, and facilities to support globally competitive programs becomes difficult. A key concern was the perception that EL undertakes projects because they are feasible, rather than because they meet a clearly defined set of needs. A well-defined strategic planning process would clarify the purpose behind each program, establish clear benchmarks to measure progress, and identify go/no-go decision points. When selecting projects, an internal management review to look across the laboratory, NIST, and the broader stakeholder community could help ensure that efforts are not duplicated and help to identify opportunities for collaboration.
Many research programs presented to the panel appeared to operate independently, with little interaction between them. This isolated approach may stem from the practice of assigning individual projects to a single principal investigator (PI), who takes sole responsibility for the project. Additionally, funding is often provided to individual PIs annually rather than to EL based on strategic initiatives, which makes fostering collaboration more difficult. EL could consider consolidating research projects that could benefit from collaboration, improving overall efficiency. Activities such as attending conferences, hosting workshops (both on and offsite), and organizing events that bring together industry and research entities could help EL allocate its limited resources more effectively. Strategic planning would also guide staff recruitment with an eye to the big picture.
For maximum impact, EL’s strategic planning would need to align with the needs of its stakeholders, including industry. It is unclear how EL determines whether its work aligns with stakeholder priorities. Without this alignment, it is challenging to assess whether EL is making the most relevant, cost-effective, and meaningful contributions. Engaging more deeply with industry and other stakeholders would provide insights into their needs, allowing EL to develop projects that directly address those challenges. This approach would enhance the relevance and effectiveness of EL’s work, fostering innovation and progress within the stakeholder communities.
Key Recommendation 1: The Engineering Laboratory (EL) should determine and describe the strategic directions it will pursue and how personnel and resources will be allocated to pursue those directions. Management reviews during project selection should use the strategic directions to avoid duplication, look for collaboration opportunities, and guide the recruiting of any new staff. EL should also develop and publish a long-term strategic plan that includes input from stakeholders such as industry.
Metrics and Stakeholder Relevance
Although EL’s mission is to “promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology for engineered systems in ways that enhance
economic security and improve quality of life,” the panel was left questioning how EL measures its success. Specifically, how does EL evaluate whether it is meeting its mission, and what metrics are used to gauge success? There is a need for clear evaluation criteria and metrics to both select projects and assess their success. The panel found no obvious criteria for evaluating how programs are selected, nor were there clear metrics to measure a program’s success or that could be used in deciding whether to maintain or terminate a program of research. Additionally, it was unclear if programs are periodically assessed for their relevance to industry stakeholders. The panel was unable to obtain detailed information on how projects are benchmarked or how key performance indicators are measured. Effectiveness and impact are best assessed at the individual project level within EL. Metrics do not need to be perfect to be useful. Some suggested metrics (a few of which may already be in place) include completion rate for project milestones, schedule compliance, software downloads, publications, citations, industry adoption of engineering technology, integration of standards, report generation, patents, licensing, and recognition through awards.
If EL is already periodically evaluating performance and program success metrics, it would be beneficial to share this track record to demonstrate the value of both past and future programs. This would allow stakeholders to appreciate the wide range of outputs from EL’s work. As NIST strives to be a world-class organization, it may benefit from further defining what “world-class” means. A clear definition would help guide EL and provide assessment panels with feedback on progress toward that goal.
The pace of research is also critical for industry relevance. EL’s typical 2–5-year project cycles align with standard industry timelines, but the rapid pace of technological advancement risks rendering long-term research outdated by the time it concludes. To better meet industry demands, EL might consider reassessing project milestones every 12 months. This would provide an opportunity to adjust milestones as needed to ensure that the work remains relevant and impactful.
Currently, EL does not seem to communicate externally the success metrics that demonstrate the impact of its research. Also, as shown in Figure 2-5, the budgets for EL’s goals have either remained stable, increased slightly over time, or decreased (the decrease shown in Figure 2-5 was due to a reorganization, with some of EL’s groups moving to the Communications Technology Laboratory). These data, however, are not corrected for inflation. This means that EL has likely either been barely maintaining purchasing power in the face of inflation or losing purchasing power. Clear metrics of the sort recommended here, and the clear communication of them to stakeholders and funders, could also be used to demonstrate EL’s value proposition to appropriators.
Key Recommendation 2: The Engineering Laboratory (EL) should communicate the current clear metrics for project adoption and success that are already in place. A management review of the work portfolio should be considered as a tool to determine whether to keep a project going or terminate it. EL should communicate the metrics externally to stakeholders and review panels. The adoption of standards by industrial groups to which EL has contributed should be externally communicated to stakeholders regularly. EL should also define what it means to be “world-class” so it can know when it is meeting this goal.
REFERENCE
NIST (National Institute of Standards and Technology). 2021. “About EL.” Updated June 2. https://www.nist.gov/el/about-el.
