2

State of the Art, Promises, and Risks of Machine Learning

2.1 EMERGING MACHINE LEARNING–ENABLED CAPABILITIES

The advent of large-scale data sets, scalable neural architectures, and increased computational power has revolutionized deep learning, unlocking unprecedented capabilities that surpass those of previous approaches in machine learning (ML) and other relevant fields. ML models and systems developed using deep learning can tackle complex problems involving perception and language, decision making and control, and interaction and collaboration with unprecedented performance. The sheer scale of modern ML models, together with the data sets used to train them, enable the synthesis and generation of intricate, contextual patterns beyond the reach of previous approaches in ML or other related disciplines. When applied to cognitive tasks, some researchers argue this process of synthesis and generation can even approximate a form of reasoning.¹

Perception and Language

Deep learning has enabled computers to measure and interpret the physical world with accuracy that was previously unattainable.² Before deep learning, computer vision and other ML relied on manual feature engineering for complex tasks such as object recognition and scene understanding. With the rise of deep learning, algorithms can

___________________

¹ E. Kiciman, R. Ness, A. Sharma, and C. Tan, 2023, “Casual Reasoning and Large Language Models: Opening a New Frontier for Causality,” arXiv preprint arXiv:2305.00050.

² A. Krizhevsky, I. Sutskever, and G.E. Hinton, 2017, “ImageNet Classification with Deep Convolutional Neural Networks,” Communications of the ACM 60(6):84–90.

now automatically learn hierarchical features from data, allowing for more accurate and robust perception systems (see Figure 2-1). These advancements have resulted in unprecedented levels of performance when perceiving the physical world. This has found applications in face recognition, automated vehicles, and medical image analysis, among other fields.

ML in natural language processing (NLP) modeling has revolutionized our interaction with computers and information. Early language models faced challenges in understanding context and semantic nuances. With the advent of recurrent neural networks

(RNNs) and recently, attention-based transformers, large language models (LLMs) can now capture intricate linguistic patterns, which has enabled the development of advanced chatbots, translation services, and sentiment analysis tools. The era of ML has ushered in an age where machines process and generate language in a more human-like manner, significantly surpassing previous capabilities.³

In the context of using ML in the physical world, the integration of computer vision and natural language processing results in vision-language models that are useful for tasks from image captioning to users able to easily describe scenarios or potential tasks to, say, robots.⁴ In Figure 2-2, one can naturally describe household cleaning tasks while respecting user preferences for where to store various objects. Popular examples of vision-language models include OpenAI GPT-4o⁵ and Google Gemini.⁶

Decision Making and Control

Traditional decision-making systems often relied on rule-based or model-based approaches that limited their adaptability to dynamic or unknown scenarios. ML has introduced models that are much more adaptable to changing conditions because they learn from data. Reinforcement learning enables systems to discover high-performing decision-making policies and rules based on trial and error, which optimizes the desired

___________________

³ A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, 2018, “Improving Language Understanding by Generative Pre-Training,” OpenAI, https://openai.com/index/language-unsupervised.

⁴ J. Wu, R. Antonova, A. Kan, et al., 2023, “TidyBot: Personalized Robot Assistance with Large Language Models,” arXiv preprint arXiv:2305.05658.

⁵ OpenAI, 2025, Introducing 4o Image Generation,”https://openai.com/index/introducing-4o-image-generation.

⁶ Gemini, “Homepage,” https://gemini.google.com, accessed April 28, 2025.

outcomes. A very well-known demonstration of this capability was AlphaGo, a system based on reinforcement learning, which beat the Go world champion.⁷ In a more recent physical demonstration shown in Figure 2-3, reinforcement learning was applied to control aerial drones in a drone racing competition.⁸ Deep reinforcement learning was not only able to improve upon previous control theoretic approaches but to also beat the Swiss drone racing champion by a half-second.

Such new ML-based capabilities that match or exceed human champions in decision making can have numerous applications in physical domains. It should be noted, however, that once the drone racetrack was changed a bit, the human champions were able to easily outperform the reinforcement learning approach drones, because the training experiences did not generalize effectively to the new environment. Similar results have been observed in car racing games.⁹

In dynamic environments, operating safely requires the ability to predict the motion of numerous objects. Predicting traffic while considering environmental conditions such as rain and snow, the proximity of construction sites, and large special events such as sports matches and concerts is challenging for any approach. Identifying objects and

___________________

⁷ D. Silver, J. Schrittwieser, K. Simonyan, et al., 2017, “Mastering the Game of Go Without Human Knowledge,” Nature 550:354–359.

⁸ Y. Song, A. Romero, M. Müller, V. Koltun, and D. Scaramuzza, 2023, “Reaching the Limit in Autonomous Racing: Optimal Control Versus Reinforcement Learning,” Science Robotics 8(82):eadf1462.

⁹ P.R. Wurman, S. Barrett, K. Kawamoto, et al., 2022, “Outracing Champion Gran Turismo Drivers with Deep Reinforcement Learning,” Nature 602:223–228.

predicting their path sufficiently far into the future is challenging and requires capturing not only the laws of physics but also social conventions. ML architectures, such as recurrent neural networks and long short-term memory (LSTM) models are being applied to time series data that are correlated across space and time. Such prediction capabilities are now critical for self-driving cars as well as numerous other applications in health care, climate, and infrastructure.¹⁰

Interaction and Collaboration

The interactive capabilities of ML have transformed human–computer interaction paradigms. Early systems struggled to adapt to user behavior and preferences. With ML, including foundational models, systems can now dynamically adjust to user inputs, providing a more personalized and responsive experience. Whether in recommendation systems, gaming, or virtual assistants, ML has ushered in an era of heightened interactivity, making technology an intuitive and integral part of daily life.

An example application of the use of ML in human–robot interaction is the robot assistive feeding system shown in Figure 2-4. Such systems in the past were designed for one type of food. Today, ML-based approaches, as described recently, can flexibly adapt to different food types while also incorporating human feedback both in the choice of the next food item as well as in the feeding approach. ML algorithms now facilitate seamless interactivity by understanding user preferences, learning from interactions, and adapting to evolving needs. This has paved the way for interactive robots, interactive design tools, and personalized user experiences across various domains.

ML brings significant advantages to collaborative, multi-agent systems, enhancing their adaptability and efficiency. Federated learning allows the privacy-preserving exchange of information with a trusted central aggregator which eventually constructs an aggregated global model and distributes it to the participating systems.¹¹ For example, local brain tumor prediction models can be shared among hospitals in order to develop a more accurate aggregate model to inform cancer treatment. Because hospitals share models and not data, this can be done while ensuring privacy of hospital data.

___________________

¹⁰ S. Ettinger, S. Cheng, B. Caine, et al., 2021, “Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset,” Pp. 9690–9699 in IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.

¹¹ S. Bakas, M. Reyes, A. Jakab, et al., 2018, “Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge,” arXiv preprint arXiv:1811.02629.

While federated learning adopts a hierarchical architecture, more distributed learning variants with peer-to-peer learning include graph neural networks (GNNs) and multi-agent reinforcement learning (MARL).^12,13,14,15

Given these transformative capabilities and the potential for immediate to long-term impact, researchers in every application domain have been studying how to develop and deploy these new ML-based capabilities. While the impact of ML is horizontal and will touch numerous fields, the next sections survey some safety-critical domains that are important for this study.

2.2 MACHINE LEARNING IN INTELLIGENT INFRASTRUCTURE

Infrastructure plays a vital role in daily life, encompassing a wide range of essential facilities and structures, including transportation systems, energy infrastructure, and food and water infrastructure, among many other domains. Such infrastructure systems constitute the cornerstone of a nation’s development and resilience, influencing economic

___________________

¹² K. Zhang, Z. Yang, and T. Başar, 2021, “Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms,” In Handbook of Reinforcement Learning and Control. Studies in Systems, Decision and Control, K.G. Vamvoudakis, Y. Wan, F.L. Lewis, D. Cansever, eds., Vol. 325, Springer, Cham.

¹³ Z. Zhou, G. Liu, and Y. Tang, 2023, “Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges,” arXiv preprint arXiv:2305.10091.

¹⁴ A. Tagliabue, K. Kondo, T. Zhao, M. Peterson, C.T. Tewari, and J.P. How, 2023, “REAL: Resilience and Adaptation Using Large Language Models on Autonomous Aerial Robots,” arXiv preprint arXiv:2311.01403.

¹⁵ T. Zhao, A. Tagliabue, and J.P. How, 2023, “Efficient Deep Learning of Robust, Adaptive Policies Using Tube MPC-Guided Data Augmentation,” arXiv preprint arXiv:2303.15688.

productivity, national security, and societal well-being on a profound scale. Despite the importance of the U.S. national infrastructure, its current condition gets the barely passing grade of C− in a 2021 report by the American Society of Civil Engineers.¹⁶

Embedded computing systems, which instrument infrastructure with sensor and actuators, yield a wealth of data to which ML can be applied to improve the safety, efficiency, and resilience of U.S. infrastructure. Such systems are known as intelligent infrastructure, defined in a 2017 report by the Computing Community Consortium¹⁷ as “the integrated sensing and data analytics with municipal capabilities and services that enable evidence-based operations and decision making.”

The potential benefits for using intelligence in infrastructure ranges across numerous infrastructure sectors, as shown in Table 2-1. Realizing this vision and the benefits shown in Table 2-1 will require numerous advances in cyber-physical systems (infrastructure sensors, computing, and actuation), networking systems, etc. The integration of ML is poised to revolutionize and elevate the intelligence of our infrastructure systems. By harnessing the power of advanced algorithms and machine learning, ML-driven analytics and decision-making processes will enhance efficiency, optimize resource utilization, detect failures and attacks, and contribute to proactive problem-solving.

Table 2-1 shows the potential of ML across infrastructure domains, while Figure 2-5 shows the potential broad impact of ML within a specific application domain—in this case the energy sector, where ML can be used in generation, transmission, predictive maintenance, and energy management at the consumer level. Similar breadth and magnitudes of impact are possible in many sectors including agriculture, smart cities, manufacturing, and transportation.¹⁸

Finding 2-1: The synergy of ML with intelligent infrastructure signifies a major step forward, promising a future where infrastructure is not just interconnected but also possesses the abilities to learn and improve performance, ultimately advancing the capabilities and resilience of the U.S. intelligent infrastructure.

___________________

¹⁶ American Society of Civil Engineers, 2021, 2021 Report Card for America’s Infrastructure: A Comprehensive Assessment of America’s Infrastructure, https://infrastructurereportcard.org.

¹⁷ E. Mynatt, J. Clark, G. Hager, et al., 2017, “A National Research Agenda for Intelligent Infrastructure,” Computing Research Association, http://cra.org/ccc/resources/ccc-led-whitepapers.

¹⁸ See, for example, J. Sipple, 2020, “Interpretable, Multidimensional, Multimodal Anomaly Detection with Negative Sampling for Detection of Device Failure,” International Conference on Machine Learning, https://doi.org/10.48550/arXiv.2007.10088; B.F. Spencer, V. Hoskere, and Y. Narazaki, 2019, “Advances in Computer Vision-Based Civil Infrastructure Inspection and Monitoring, Engineering 5(2):199–222; K. Borden, M. Huntington, M. Kamat, A. Singla, J. Wijpkema, and B. Wiseman, 2022, “The Future Is Now: Unlocking the Promise of AI in Industrials,” McKinsey & Company, December 6, https://www.mckinsey.com/industries/automotive-andassembly/our-insights/the-future-is-now-unlocking-the-promise-of-ai-in-industrials; A. Kuhlmann, E. Mehlum, and J. Moore, 2021, “Harnessing Artificial Intelligence to Accelerate the Energy Transition,” White paper, World Economic Forum, September, https://www3.weforum.org/docs/WEF_Harnessing_AI_to_accelerate_the_Energy_Transition_2021.pdf; Microsoft, “Project FarmVibes – Microsoft Research: Overview,” https://www.microsoft.com/en-us/research/project/project-farmvibes, accessed October 4, 2025.

TABLE 2-1 Benefits of Intelligent Infrastructure Across Domains

2.3 MACHINE LEARNING IN HEALTH CARE

Recent advances in ML are seeing broad exploration and adoption across health care and medicine. This section provides a brief overview of emerging ML applications in robot-assisted medical procedures and language-based interaction. ML has the potential to produce many new capabilities in health care and medicine spanning perception and language, planning and control, and collaboration.

Due to current and further expected shortages of skilled medical personnel along with increased need for expert and invasive medical treatments, automated surgical robotic systems have gained increased attention. The field is quickly advancing from

theoretical demonstration to providing robotic platforms with onboard learning models. For example, researchers recently demonstrated flexible, automatically steered needles in vivo in a porcine.^19,20

The complex nature of invasive surgery has focused ML applications for surgical robotics on interpreting images and videos. While end-to-end surgical solutions are being explored, ML is primarily applied to a specific portion of the process, including identifying anatomical structures (e.g., organs), identifying and tracking instrument usage by surgeons, classifying phases of surgeries for task-level breakdown, predicting the time window required for a surgery both for pre-scheduling and for real-time adaptation, and for learning a specific skill—for example, identifying an incision-line or suturing a cut.

Safety for surgical robotics is typically addressed by the development of systems that encourage or require expert supervision or collaboration. Levels of autonomy for surgical robotics have been proposed that mirror those for autonomous driving systems with some modifications for the prescriptive and systematic task-level processes that medical surgery requires.^21,22 Additionally, some of the most promising applications of learning in surgical robotics aims to enhance surgical skill and monitor human stress levels in order to provide interventions to enhance patient outcomes. This has been proposed to be used for offline, individual training and for real-time application.

The most recent advancements in generative pre-trained models have sparked excitement about their potential applications in health care and medicine. The types of interactions being proposed for use with ML chat systems are broad and include, for example, medical note-taking, medical question answering, and medical consultation.²³ While initial applications of generative pre-trained models have focused on language-based interactions, multimodal models are expected to enable multimedia interactions with ML agents incorporating text, imaging, genomics, and many other data modalities.^24,25 Advancements in multimodal generative models and related technologies, such as vision-language and language-action models, may provide further transformative capabilities

___________________

¹⁹ H. Saeidi, J.D. Opfermann, M. Kam, et al., 2022, “Autonomous Robotic Laparoscopic Surgery for Intestinal Anastomosis,” Science Robotics 7(62).

²⁰ A. Kuntz, M. Emerson, T.E. Ertop, et al., 2023, “Autonomous Medical Needle Steering In Vivo,” Science Robotics 8(82):eadf7614.

²¹ J. Han, J. Davids, H. Ashrafian, A. Darzi, D.S. Elson, and M. Sodergren, 2022, “A Systematic Review of Robotic Surgery: From Supervised Paradigms to Fully Autonomous Robotic Approaches,” The International Journal of Medical Robotics and Computer Assisted Surgery 18(2):e2358.

²² G.-Z. Yang, J. Cambias, K. Cleary, et al., 2017, “Medical Robotics—Regulatory, Ethical, and Legal Considerations for Increasing Levels of Autonomy,” Science Robotics 2(4).

²³ P. Lee, S. Bubeck, and J. Petro, 2023, “Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine,” New England Journal of Medicine 388(13):1233–1239.

²⁴ J.N. Acosta, G.J. Falcone, P. Rajpurkar, and E.J. Topol, 2022, “Multimodal Biomedical AI,” Nature Medicine 28:1773–1784.

²⁵ E.J. Topol, 2023, “As Artificial Intelligence Goes Multimodal, Medical Applications Multiply,” Science 381:eadk6139.

in health-related applications involving perception, planning, and control, such as the previously described robot-assisted medical procedures.

Initial studies of the application of general-knowledge ML systems for medical applications illustrate both the promise and the potential risks.²⁶ For example, ML agents have confidently provided false answers—for example, by providing non-existent information in a medical summary. Ongoing research seeks to address known limitations of generative models, such as hallucinations. Recent work has, for example, demonstrated how domain-specialization^27,28 and sophisticated prompting strategies²⁹ can improve performance on health-related tasks and have demonstrated the promise of ML systems as evaluators of content, helping to find and correct errors or provide additional perspectives.³⁰

Finding 2-2: ML, including recent advancements in foundation models, promises to transform health care and medicine by enabling new applications ranging from robot-assisted medical procedures to language-based interaction with AI chat systems. These applications have the potential to improve patient outcomes, reduce costs, and lower barriers to expert care.

2.4 MACHINE LEARNING IN MANUFACTURING

There have been persistent indicators of difficulties hiring in the manufacturing sector—for example, the Bureau of Labor Statistics reports around 482,000 manufacturing jobs opening in February 2025.³¹ While industrial control systems and, more recently, special-purpose robots have proliferated in manufacturing, human labor is still needed for tasks that require physical dexterity and cognitive flexibility, such as those requiring complex assembly or adaptation to novel situations.³² Shortages of workers willing and able to

___________________

²⁶ D. Dash, E. Horvitz, and N. Shah, 2023, “How Well Do Large Language Models Support Clinician Information Needs?” Stanford University Human-Centered Artificial Intelligence, https://hai.stanford.edu/news/how-well-do-large-language-models-support-clinician-information-needs.

²⁷ T. Tu, S. Azizi, D. Driess, et al., 2023, “Towards Generalist Biomedical AI,” arXiv preprint arXiv:2307.14334.

²⁸ C. Li, C. Wong, S. Zhang, et al., 2023, “LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day,” arXiv preprint arXiv:2306.00890.

²⁹ H. Nori, Y.T. Lee, S. Zhang, et al., 2023, “Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine,” arXiv preprint arXiv:2311.16452.

³⁰ Z. Gero, C. Singh, H. Cheng, et al., 2023, “Self-Verification Improves Few-Shot Clinical Information Extraction,” arXiv preprint arXiv:2306.00024.

³¹ U.S. Department of Labor, 2025, “Job Openings and Labor Turnover–November 2024–Table 1,” Bureau of Labor Statistics, January 7, Table 1, Job Openings Levels and Rates by Industry and Region, Seasonally Adjusted - 2023 M11 Results, www.bls.gov/jlt.

³² G. Zeba, M. Dabić, M. Čičak, T. Daim, and H. Yalcin, 2021, “Technology Mining: Artificial Intelligence in Manufacturing,” Technological Forecasting and Social Change 171:120971.

perform these potentially hazardous roles have helped to motivate increased investment in general-purpose robots capable of human-like dexterity and adaptability and able to collaborate effectively with human workers to perform tasks that have traditionally been difficult to fully automate.³³ While production deployment of general-purpose robots in manufacturing has been limited to date,³⁴ recent advancements in robotics and ML have led to new investments in “human-centric” robots.³⁵

ML is already broadly applied across the spectrum of robotics applications for tasks that involve sensing, localization, motion coordination, high-level task planning, and information fusion.³⁶ Even so, many challenges remain for ML in robotics applications that demand human-like dexterity and reasoning such as those currently performed by humans in manufacturing. Advances in robotics-oriented foundation models are beginning to show promise in addressing these challenges. Recent work has demonstrated the use of foundation models for robotics tasks spanning perception, decision making and control in experimental settings such as open-vocabulary scene segmentation and semantic task decomposition.³⁷ Further advances in areas such as real-time task execution, safety guarantees, and effective human–machine collaboration represent vital research frontiers in this space.

Ensuring safe deployment will be paramount in unlocking the value of general-purpose robots in manufacturing. Some system concepts incorporate human oversight and monitoring as integral components to safe execution.³⁸ However, studies have shown that human monitoring breaks down as task complexity and the number of robots being supervised increases.³⁹ Formal control theory has been combined with ML to provide guardrails on ML-based controllers. However, the complexity of most robotics problems limits the applicability of such techniques. Providing explainable and legible robotic behaviors extracted from learned controllers is another approach to safety. Although the resulting behaviors are understandable, they do not necessarily ensure safe operations. Increased investment toward creating production-ready general-purpose robots that can safely and effectively harness these emerging capabilities could unlock

___________________

³³ A.A. Malik and A. Brem, 2021, “Digital Twins for Collaborative Robots: A Case Study in Human–Robot Interaction,” Robotics and Computer-Integrated Manufacturing 68:102092.

³⁴ A.A. Malik, T. Masood, and A. Brem, 2023, “Intelligent Humanoids in Manufacturing to Address Worker Shortage and Skill Gaps: Case of Tesla Optimus,” arXiv preprint arXiv:2304.04949.

³⁵ M. O’Brien, 2023, “Robot Startups See Huge Market in Replacing Human Workers: ‘We Can Sell Millions of Humanoids, Billions Maybe,’” The Associated Press, November 5. https://fortune.com/2023/11/05/when-will-robots-replace-humans-startups-elon-musk-humanoids-optimus.

³⁶ L. Wang, R. Gao, J. Váncza, et al., 2019, “Symbiotic Human–Robot Collaborative Assembly,” CIRP Annals 68(2):701–726.

³⁷ R. Firoozi, J. Tucker, S. Tian, et al., 2023, “Foundation Models in Robotics: Applications, Challenges and the Future,” The International Journal of Robotics Research, https://doi.org/10.1177/02783649241281508.

³⁸ L. Wang, R. Gao, J. Váncza, et al., 2019, “Symbiotic Human–Robot Collaborative Assembly,” CIRP Annals 68(2):701–726.

³⁹ National Academies of Sciences, Engineering, and Medicine, 2022, Human-AI Teaming: State-of-the-Art and Research Needs, National Academies Press, https://doi.org/10.17226/26355.

tremendous value across manufacturing sectors. This should include investment in work on human–computer and human–robot interaction.

Finding 2-3: Safe deployment of general-purpose robotic systems capable of human-like dexterity and adaptability and capable of effectively collaborating with human workers to perform tasks could unlock tremendous value across manufacturing sectors, reducing hazardous exposure for workers and helping to address shortages in skilled human labor. Advances in ML, and foundation models in particular, are beginning to show promise in enabling key use cases, but areas such as real-time task execution, safety guarantees, and effective human–machine collaboration represent vital research frontiers toward realizing that value.

2.5 MACHINE LEARNING IN AUTOMOTIVE SYSTEMS

ML is already a key technology in automotive systems. The range of ML-supported applications in advanced driver assistance systems include collision prediction and warning, lane keeping control, lane change assistance, and many more. While these are all integrated in today’s SAE Level 2 vehicles (where the driver is required to assume control when needed), advances in ML pave the path to higher degrees of automation and ultimately may allow fully autonomous vehicles (SAE Levels 4 and 5).⁴⁰ Market demands for higher levels of automation include freeing the driver to carry out other activities, increasing vehicle safety, improved mobility for the disabled, and cutting the costs of freight and public transportation services. ML technology is also employed in vehicle production processes, such as quality monitoring, and in predictive maintenance.⁴¹

The global market for automobiles with ML capabilities is expected to continue growing in the coming years. Consumers are welcoming a new range of automated features, ranging from assisted features to full self-driving. By 2030, up to 10 percent of global new car sales could be with Level 3 automation.⁴²

___________________

⁴⁰ SAE International, 2021, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” SAE J3016_202104, April 30, https://www.sae.org/standards/content/j3016_202104.

⁴¹ Mercedes-Benz, 2023, “Automated Driving Revolution: Mercedes-Benz Announces U.S. Availability of DRIVE PILOT—The World’s First Certified SAE Level 3 System for the U.S. Market,” September 27, https://media.mbusa.com/releases/automated-driving-revolution-mercedes-benz-announces-us-availability-of-drive-pilot-the-worlds-first-certified-sae-level-3-system-for-the-us-market.

⁴² Goldman-Sachs, “Partially Autonomous Cars Forecast to Comprise 10% of New Vehicle Sales by 2030,” https://www.goldmansachs.com/insights/articles/partially-autonomous-cars-forecast-to-comprise-10-percent-of-new-vehicle-sales-by-2030, accessed April 28, 2025.

In the United States, Mercedes offers certified Level 3 autonomous driving capability (currently available in California and Nevada),⁴³ which allows the car to drive itself in certain circumstances. This is in contrast to Tesla’s Level 2 autopilot, which permits the driver to take their hands off the wheel and feet off the pedals, while remaining alert and ready to take control at any time.

On a more technical level, deep learning methods are shown to be effective in fusing multi-modal sensors for scene understanding. Figure 2-6 shows several published methods and their performance on a vision test set developed at the Karlsruhe Institute of Technology and the Toyota Technological Institute (KITTI).⁴⁴ A survey article by Feng et al. discusses the state of the art in using deep learning in perception systems in vehicles.⁴⁵

___________________

⁴³ Mercedes-Benz, “Drive Pilot,” https://www.mbusa.com/en/owners/manuals/drive-pilot, accessed April 28, 2025.

⁴⁴ A. Geiger, P. Lenz, and R. Urtasun, 2012, “Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite,” 2012 IEEE Conference on Computer Vision and Pattern Recognition 3354–3361.

⁴⁵ D. Feng, C. Haase-Schütz, L. Rosenbaum, et al., 2021, “Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges,” IEEE Transactions on Intelligent Transportation Systems 22:1341–1360.

Finding 2-4: ML is already, today, an indispensable technology in the automotive industry. It is key to enabling future systems to continually move to higher levels of automation.

2.6 EMERGING RISKS FOR MACHINE LEARNING IN SAFETY-CRITICAL APPLICATIONS

Integrating ML with physical systems such as robots, energy grids, and medical devices promises unprecedented efficiency, autonomy, and functionality advancements. The potential benefits are vast—for example, autonomous surgical robots performing intricate procedures and ML-driven energy systems optimizing electrical power distribution. However, this fusion also introduces a new spectrum of previously non-existent or minimal risks in traditional systems. Understanding these risks is crucial for developers, policymakers, and users to ensure that deploying ML-enabled physical systems is safe and beneficial to humanity.

One of the most immediate concerns is the potential for physical harm resulting from ML errors. A malfunction or misinterpretation of data could lead to accidents, causing injury or damage. For example, an autonomous robot in a manufacturing plant might misidentify a human worker as an object to be moved, leading to dangerous interactions.

In 2018, an autonomous Uber SUV struck and killed a pedestrian in Tempe, Arizona. The National Transportation Safety Board concluded that the vehicle’s ML system (i.e., the computer vision systems) failed to correctly identify the pedestrian walking a bicycle across the road at night.⁴⁶ The perception system oscillated among classifying the pedestrian as a vehicle, a bicycle, and an unknown object, leading to a delay in initiating braking. This tragic incident highlighted how ML perception errors can have fatal consequences when controlling vehicles or other physical systems. Since 2021, the National Highway Traffic Safety Administration has required self-driving vehicle companies to report all crashes. Since then, there have been more than 3,000 reported crashes.⁴⁷

Another domain where ML errors can result in the loss of human life is using ML for data-driven diagnostics. ML’s role would typically be to interpret the data coming from sensors (MRI, wearables, medical devices) to diagnose a medical condition. However, the various sources of error in the ML methods could lead to serious consequences, as shown in Figure 2-7.

___________________

⁴⁶ NTSB, 2018, “Highway Accident Report Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian,” Tempe, AZ, March 18.

⁴⁷ National Highway Traffic Safety Administration, “Standing General Order on Crash Reporting,” https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting, accessed April 28, 2025.

Even with access to large, high-quality data sets, ML-enabled technologies in clinical practice face at least three major sources of error. First, ML predictions can be significantly affected by noise in input data during use—for example, ultrasound scans are prone to errors due to operator inexperience, patient cooperation, and clinical context, which can adversely affect ML outcomes. Second, misclassifications may occur due to data set shifts. In this common issue, slight differences between the training and real-world clinical data—stemming from variations in population groups, hospital protocols, or equipment from different manufacturers—can lead to decreased accuracy. Studies have shown that ML models trained on specific MRI scanners lose accuracy when applied to data from other machines. ML systems for diagnosing conditions like pneumonia or retinal diseases perform poorly when tested on data from different hospitals or devices. These challenges illustrate the difficulty in developing AI tools that consistently maintain high accuracy across diverse populations, clinical settings, and equipment variations.

The integration of ML with physical systems also amplifies security vulnerabilities. ML systems are susceptible to hacking, spoofing, and other cyberattacks that can manipulate their behavior. In the context of autonomous vehicles or drones, a compromised ML system could be redirected to cause collisions or invade restricted airspace. Energy grids controlled by ML could be targeted to disrupt power supplies, which could affect hospitals, communication networks, and other essential services.

Adversarial attacks involve manipulating input data to deceive ML models into making incorrect decisions without altering the ML system itself. These attacks exploit how ML models interpret data, often by introducing subtle perturbations that are imperceptible to humans but significantly impact the ML output. In 2018, researchers demonstrated that small alterations to road signs could mislead autonomous vehicle ML

systems. Placing strategically designed stickers on stop signs caused the ML model to misclassify them as speed limit signs.⁴⁸

The potential for adversarial examples to fool ML models has, for example, been demonstrated in medical imaging, where security researchers have demonstrated how altering scans in ways that are invisible to radiologists can cause ML models to misdiagnose. An ML system might miss a cancerous tumor or falsely identify a healthy organ as diseased (see Figure 2-8).

A new safety risk emerged with the advent of generative AI and LLMs such as OpenAI’s GPT and Anthropic’s Claude. Such models have a variety of safeguards, including input filtering and safety alignment using model finetuning. Safety alignment involves ensuring that these systems behave in ways that are consistent with human values and safety requirements, preventing unintended or harmful actions. This safety effort focuses on aligning the model’s goals and behaviors with a set of ethical standards and regulatory guidelines in an effort to mitigate risks associated with autonomous decision making.

Recently, a new risk known as jailbreaking has created new safety and security concerns for generative models. LLM jailbreaking refers to bypassing the built-in safety features and content filters. Malicious users may attempt to manipulate the model into producing unsafe content through specific prompts or techniques. Jailbreaking poses significant security, ethical, and societal risks, as it can result in harmful information,

___________________

⁴⁸ K. Eykholt, I. Evtimov, E. Fernandes, et al., 2018, “Robust Physical-World Attacks on Deep Learning Visual Classification,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

violate privacy, and erode trust in AI systems. As LLMs are now integrated not only into research robots but also into commercial products, this creates a new safety concern that can result in harm in the physical world (see Figure 2-9).

These are some but not all risks that are caused by ML algorithms in AI-enabled systems.

Finding 2-5: While application of ML has resulted in observable benefits in society, there are numerous safety risks that emerge when ML is integrated into real systems. These risks occur across many different domains and can result in loss of life and damage to property or the physical environment.

Addressing these challenges will ultimately lead to the safe and robust integration of ML into physical systems, thereby increasing trustworthiness and enabling these technologies to deliver their full potential benefits across various sectors. Doing so will require collaboration among researchers, industry professionals, and policymakers to develop solutions that enhance the safety and reliability of ML-based technologies.

Finding 2-6: Although ML now matches or exceeds human performance in many tasks, its deployment in physical systems presents both opportunities and risks. The opportunities include novel applications, enhanced efficiency, and improved human–machine collaboration, while the risks include classification errors and vulnerability to malicious attacks, among others.