Machine Learning for Safety-Critical Applications: Opportunities, Challenges, and a Research Agenda (2025)
Chapter: Appendix A: Statement of Task
Visit
NAP.edu/10766
to get more information about this book, to buy it in print, or to
download it as a free PDF.
Previous Chapter:
5 Societal Considerations to Build Public Understanding and Confidence in Safety-Critical Systems with Machine Learning Components
A
Next Chapter: Appendix B: Briefings to the Committee
Suggested Citation:
"Appendix A: Statement of Task." National Academies of Sciences, Engineering, and Medicine. 2025. Machine Learning for Safety-Critical Applications: Opportunities, Challenges, and a Research Agenda. Washington, DC: The National Academies Press.
doi: 10.17226/27970.
A
Statement of Task
A National Academies of Sciences, Engineering, and Medicine study will explore the trustworthiness of machine learning (ML), especially very large or complex models, in safety-critical applications. The study will consider such questions as:
- What are core principles of trustworthiness in safety-critical systems? What adaptations are needed to accommodate ML models?
- What metrics of trustworthiness are currently used to assess safety-critical systems that do not rely on ML? Which are applicable to systems that rely on ML and how can they best be applied? What new metrics are needed for systems that rely on ML? For example, what does it mean for a system that relies on ML to be correct?
- How should systems relying on one or more ML models be tested and evaluated? What types of assurances are possible? How can reliability requirements be satisfied when a system employs nontransparent models? What impact do limits on the explainability of ML models have on evaluating and ensuring safety?
- How does the robustness of today’s ML models compare with the level of robustness expected in non-ML systems that are certified or otherwise approved for safety-critical applications? What are opportunities to better understand and enhance the robustness of ML models?
- How do traditional approaches for achieving trustworthiness such as testing need to be modified for safety-critical systems that rely on ML? What new formalisms are needed to describe and assess ML trustworthiness?
Suggested Citation:
"Appendix A: Statement of Task." National Academies of Sciences, Engineering, and Medicine. 2025. Machine Learning for Safety-Critical Applications: Opportunities, Challenges, and a Research Agenda. Washington, DC: The National Academies Press.
doi: 10.17226/27970.
- How can monitoring and run-time verification be used with systems that rely on ML to identify potentially unsafe conditions and enable a system or its human operators to take fail-safe action?
- What investments in research on ML in safety-critical systems are needed to complement ongoing work on related topics in the ML research community?
- How can the cost of ML failures in safety-critical systems be quantified and measured?
The committee’s report will describe the present state of the art in approaches to engineering safety-critical systems (both involving ML and not) and identify research that would (1) enhance understanding of the challenges in building safe systems that rely on ML and (2) foster improvements to the safety of systems that rely on ML to be improved. It may provide findings and conclusions as appropriate but will not provide recommendations.
Suggested Citation:
"Appendix A: Statement of Task." National Academies of Sciences, Engineering, and Medicine. 2025. Machine Learning for Safety-Critical Applications: Opportunities, Challenges, and a Research Agenda. Washington, DC: The National Academies Press.
doi: 10.17226/27970.
Suggested Citation:
"Appendix A: Statement of Task." National Academies of Sciences, Engineering, and Medicine. 2025. Machine Learning for Safety-Critical Applications: Opportunities, Challenges, and a Research Agenda. Washington, DC: The National Academies Press.
doi: 10.17226/27970.
Subscribe to Email from the National Academies
Keep up with all of the activities, publications, and events by subscribing to free updates by email.