5
TRANSLATION OF THE AI CODE OF CONDUCT FRAMEWORK TO REAL-WORLD APPLICATIONS

A central goal of establishing the AI Code of Conduct (AICC) framework is to ensure that the benefits of artificial intelligence (AI) in health, health care, and biomedical science are realized, and the risks mitigated. To reflect an intention of field alignment rather than a one-size-fits-all set of rules, the AICC Code Principles and Code Commitments are intentionally high level, providing guideposts for actors in the health, health care, and biomedical science system. However, to realize the goal of improved health of the U.S. population using AI, it is essential to translate the AICC Commitments into actionable, real-world practice, bringing the AICC Commitments to the next level of granularity by providing context and examples which are intended to be used to support the development of more comprehensive implementation guides by industry sectors and individual organizations. To that end, the AICC steering committee members convened expert working groups to consider and characterize real-world activities and stakeholder perspectives, responsibilities, and needs in the context of the Code Commitments and the AICC AI lifecycle. Table 5-1 provides description of the stakeholder groups directly involved in or subject to the development, use, and evaluation of health AI. It should be noted that some of the content is duplicated across stakeholder perspectives as it is anticipated that readers may choose to focus on stakeholder perspective content most relevant to themselves.

AI DEVELOPERS’ PERSPECTIVE

Developers shoulder immense responsibility when creating AI solutions that impact human health, and conceptually these communities exist as both individuals and organizations. The AICC Commitments warrant consideration at every stage of the AI lifecycle, including algorithm development, and deployment

TABLE 5-1 | Description of Key Stakeholder Groups

process, prioritizing patient alignment, security, and ethical considerations over commercial interests or individual gains. Key considerations for action mapped to the Commitments, as applicable, are discussed below. In particular, it is essential that organizations recognize potential conflicts between developing profitable AI and aligning with the Commitments and seek to prioritize the Commitments while still meeting business needs.

To that end, as described in the AI Development Lifecycle section of this chapter, developers should engage in a comprehensive impact assessment (Jacob et al., 2025) for new and existing products. Additionally, near-, short-, and long-term potential benefits and harms across the various involved parties in the value chain throughout the AI lifecycle should be identified, weighed, and mitigated where possible (Attard-Frost and Widder, 2024), and transparently shared with users of AI.

Commitment 1: Advance Humanity

Developers translate the societal, cultural, and individual goals of health and health care from other AI stakeholder groups into technical implementations that prioritize human agency. Developers also can play a strong role in fostering human connections by carefully designing human-computer interactions to optimize human communication. In health and health care, technical and algorithmic design choices will best advance humanity when aligned directly with individual goals and family priorities and when developers proactively work to mitigate misalignment in that respect (The Light Collective, 2024). In addition, it is important that developers consider how AI may be used downstream to

design mechanisms for patients and users to be aware of what (and how) the AI is operating, making transparent whether it is appropriate to be used with regard to each observation, patient, or use.

Commitment 2: Ensure Equity

Developers play a central role in ensuring that bias is assessed and mitigated, and fairness is promoted within the context of use of AI applications and are also responsible for translating the societal and cultural concepts of equity from users and receipts of AI use into technical implementations. There are ethical considerations for developers throughout the AI implementation lifecycle. As developers may not have formal ethical training and in most cases will not be able to represent the interests, culture, and preferences of end users and patients, inclusion of these groups as well as experts in ethics and equity can help guide the development process, ensuring that an AI solution’s primary aim is to advance human health. This is because it is critical to proactively identify bias starting with the initial design and conception, and this can be done by incorporating diverse perspectives, ensuring representative datasets, and implementing fairness metrics that are evaluated up front as well as on an ongoing basis (Consumer Technology Association, 2023; Echo Wang et al., 2022).

One of the cornerstones of ensuring equity and fairness and mitigating bias is for developers to prioritize and use data in AI development that are provided within clear governance frameworks that respect patient privacy, promote data equity, and seek to ensure high-quality, diverse, equitable, and representative datasets. It is also important for developers to consider and address AI models’ technical proficiency and security, as well as ethical soundness and clinical relevance—partnering with stakeholders managing the environment of use, such as health care systems, and leveraging insights from federal agencies. Last, developers can require, facilitate, and recommend to users a thorough risk assessment and benefit measurement approach that includes continuous monitoring and evaluation of AI systems to identify and mitigate potential risks while optimizing benefits for all parties that include health care providers, patients, and others.

Commitment 3: Engage Impacted Individuals

Moving beyond technology, developers must embrace a people-centered approach that prioritizes collaboration, transparency, and shared decision making. Engaging people as partners with agency throughout the AI lifecycle is crucial for developing responsible and effective AI solutions. For developers, this directly and

clearly translates to identifying and promoting engagement, representation, input, and feedback from any constituent group involved in the conceptualization, development, use, or receipt of actions or decisions from AI use throughout the entire AI lifecycle. Collaborating with these groups (e.g., patients, clinicians, and administrators) and seeking their input and feedback is foundational to ensure solutions align with their needs and expectations. These needs include concepts of fiduciary responsibility, legal liability, goals of care or desired outcomes, and concerns or risk tolerance. It should also be noted that for any AI systems where individuals are patients and recipients of AI output, prioritizing their representation throughout the development process takes on additional urgency, as does actively seeking their input and ensuring that their interests are advanced and protected.

It is also important for developers to consider the conflicting needs of multiple stakeholders, as AI products will provide the greatest benefit to humanity only if they balance the needs and preferences of multiple parties. This is of particular importance when users and the recipients of AI outputs or recommendations are not the same individual. While openly acknowledging and addressing potential conflicts among all stakeholders, the prime driver must remain the prioritization of equity, well-being, and health care advancement. This includes embracing the tension between different needs as a catalyst for critical reflection, innovation, and improvement. Embracing a continuous improvement mindset to ensure solutions remain effective and responsive is needed.

Lastly, developers should consider that they are themselves important stakeholders in the overall AI ecosystem, and that they should seek to contribute and engage as stakeholders in upstream activities that impact the AI implementation lifecycle broadly. Developers have vast experience to share in developing frameworks, standards, foundational sociotechnical constructs, and their participation in these activities would be valuable for building baseline methods and practices for the whole ecosystem.

Commitment 4: Improve Workforce Well-being

Developers play an important role in conceptualizing, developing, and implementing software solutions that improve workflows and reduce inefficiency and cognitive burden. This group plays a critical role of translation in understanding the objectives and needs of health care users and recipients of AI tools to optimize the richness, quality, and safety of human interactions in health care delivery. To accomplish this, developers must work with and engage the workforce of both health professionals themselves and health care well-being experts who can translate knowledge and best practices through user preferences. This can result in

valuable input about the impact that AI solutions might have on clinical workflows and practitioner morale and provide opportunities for optimization. Workflow may be optimized by aligning clinician preferences regarding when, where, and how outputs of the AI solutions should be displayed and shared with the internal technology teams for optimal integration into workflows (Chen et al., 2022).

Lastly, a critical challenge in ensuring satisfaction for any workflow process change is to appropriately support communication and education in its use and provide proactive plans for implementation by clinical and operational staff. While this is not generally performed by the developer, including guidelines and best practices for how organizations may most effectively implement their AI solution is likely to improve adoption and user satisfaction. These guidelines should also include any required clinician and patient disclosure statements about the use of the AI solutions, and those must be understandable, actionable, and clearly defined.

Commitment 5: Monitor Performance

Transparency builds trust and ensures responsible AI development. A foundational issue for developers is the need to intentionally design in a way that will increase the user’s trust in AI. There are several ways in which developers can promote transparency and support the maintenance of performance over time. This includes committing to monitoring AI performance and sharing information about its impact on health and safety.

First, developers should maintain appropriate transparency regarding data sources, including demographics, location, and time period of data collection so that trust can be built on the quality of the solution and the applicability to the current context (system and patient population). Understanding the data sources for AI is critical in understanding the allowable context of use and establishing appropriate performance metrics.

Developers can play an important role in promoting trust and facilitating informed decision making by relevant stakeholders by proactively developing monitoring platforms and openly disclosing product performance metrics and any specific implementation requirements, as well as feedback loops and mechanisms to support post-implementation monitoring, recalibration, and decommissioning. Developers should promote processes for monitoring, feedback, and improvement, drawing from various disciplines and expertise to ensure diverse perspectives are considered (Davis et al., 2024). Robust post-market monitoring systems with clear channels for user feedback (including patients and providers) and reporting of concerns (Saria, 2022; Vasey et al., 2022) provide a mechanism to ensure that health AI systems continue to achieve their stated goals over time.

During the development process and initial use piloting phases, root cause analyses of errors associated with a harm should be conducted without fear of retaliation because the process is critical to allow developers to learn and improve. Existing examples include patient safety organizations and coordinated vulnerability disclosure (cybersecurity) (Householder et al., 2017). From these activities, developers should also provide guidelines for characterization of harms and errors as well as remediation process recommendations for anticipated failure modes to downstream users. Additionally, proactive planning can be performed for unintentional information disclosure, considering the needs of affected parties and generating recommendations for tailoring communication strategies accordingly. Lastly, this stakeholder group should collaborate with researchers and federal agencies to promote standards for necessary oversight and support for adapting AI solutions to emerging evidence and shifting patient needs.

Commitment 6: Innovate and Learn

The developer community plays a crucial role in ensuring responsible and ethical AI innovation that enables and enhances clinicians’ ability to advance clinical practice and ensuring that AI is intentionally designed for scalability. Ongoing exploration of new approaches and incorporation of diverse perspectives are needed to ensure the robustness and reliability of AI models and solutions (Saria, 2022). This requires active participation, collaboration, and a dedication to continuous learning. Staying at the forefront of AI advancements through exploration, research, and knowledge sharing within their community, developers can contribute to the ongoing improvement of health care delivery. It is critical for developers to actively seek feedback and incorporate insights into new product design as well as iterations of deployed AI models (Saria, 2022; Vasey et al., 2022). Open communication and collaboration with stakeholders will be crucial to navigate potential misalignments and ensure that solutions are ethically sound and beneficial to all.

For most use cases, AI solutions are likely to augment users. In these scenarios, a clear teaming model should consider user roles, collaborations (how user(s) collaborate with and leverage the software to complete a task), and accountabilities (for what each party is accountable). Additionally, interfaces that are transparent and intelligible and enable each party to responsibly perform their intended role are advantageous (Henry et al., 2022).

As larger, more sensitive datasets are used in health AI, it will be paramount to continue to develop methods to support the use of unbiased, real-world data, while ensuring data provenance and carefully considering data linkage to

protect patient privacy and align with user groups’ interests. Additionally, ongoing investment in the innovation in synthetic data creation is likely to preserve privacy while promoting fair and representative data availability (Li et al., 2023).

Developers should learn from successes and continued challenges as processes for monitoring, feedback, and improvement are implemented, and they should continue to innovate in collaboration with various disciplines and expertise to promote more equitable and fairer algorithmic performance for both statistical and clinical metrics of importance (Davis et al., 2024).

By embracing the AICC Commitments and prioritizing ethical considerations, the developer community can drive responsible AI innovation that empowers health care professionals to advance the standard of clinical practice and improve patient outcomes, while simultaneously building trust among clinicians and the public and advancing their commercial interests. This requires a dedication to understanding user needs, fostering collaboration, ensuring data integrity, and continuously striving for improvement. Through these efforts, developers can contribute to a future where AI plays a transformative role in health care while upholding the highest ethical standards.

RESEARCHERS’ PERSPECTIVE

Researchers have a responsibility in both the ethical conceptualization and conduct of research but also in the application of their research. Research has driven many innovations but also can lead, and has led, to significant harms in the absence of adherence to an ethical code, such as that which occurred within the Tuskegee Study of Untreated Syphilis (CDC, n.d.). All research is now governed by “the principles of respect for persons, beneficence, and justice” outlined in the Belmont Report (HHS, 1979). These principles for human subject protection continue to improve and evolve (O’Sullivan et al., 2020). A community-accepted AI code of conduct framework is important in the research domain when considering questions relevant to the design, use, and impact of AI.

In the context of health AI, there are two major areas of relevant research: (1) AI methods and their validation (basic research), and (2) assessment of AI’s impact in the lab or in real-world settings (applied research). First, AI tools are increasingly being used to conduct research, supporting researchers in reviewing the literature, drafting portions of manuscripts, and writing analytical code (Koller et al., 2024). Second, researchers are studying how to develop, adapt, implement, and sustain AI systems, tools, and applications to support the goals and objectives of patients, clinicians, and health care organizations. Researchers must continue to adhere to ethical standards, ensure internal and external validity,

be transparent regarding assumptions and limitations, and assess and mitigate inequitable performance (NIH, 2024), as well as follow open science guidelines. AI presents novel opportunities and challenges, requiring consideration beyond existing ethical frameworks. These issues are presented below, organized within the AICC Commitment framework.

Commitment 1: Advance Humanity

AI findings and tools developed by researchers are increasingly being employed in clinical and non-clinical environments, thereby directly impacting human health and placing a greater need to consider their unintended consequences. Accordingly, potential impacts—good and bad—are of greater magnitude than in research without AI due to its immense scope and breakneck pace of implementation. The concepts of beneficence and need to avoid harm, taken from ethical frameworks, require more robust engagement with patients to develop appropriate methods of shared governance and consent.

Commitment 2: Ensure Equity

There are myriad aspects to equity warranting further research and innovation. Most importantly, researchers should consider equity and inclusion from conceptualization to execution, evaluation of mechanisms for equitable implementation, and considerations for how and when technologies or tools should be retired. Through this lens, key aspects of ensuring equity through research emerge.

Diverse data sources are essential to ensure equitable health AI benefits. Programs, such as Bridge2AI,¹ that enable ethically sourced data to train AI models—before they are applied in care—are crucial to create a virtuous learning cycle. This also includes innovation and appropriate protections in data access and sharing. While requirements are in place for sharing of data used for research, multiple studies have found that a minority of research benefits from shared data. This extends to data for development and use in health AI; additional research is warranted to maximize data synthesis (Gonzales et al., 2023; Guillaudeux et al., 2023), data de-identification and sharing along the themes of societal benefits, distribution of risks, benefits, and burdens, respect for persons, and public trust and engagement (Kalkman et al., 2019).

There are significant challenges in ensuring that AI technologies, once implemented, achieve sustained performance and avoidance of sub-population

___________________

¹ See https://bridge2ai.org (accessed June 21, 2024).

bias. This is an important area of innovation need and opportunity to provide methods and workflows to support continuous monitoring for both AI algorithmic performance and data fitness for purpose. All AI should be monitored for performance (including AI-driven monitors), innovation in automation is critical to provide coverage and scalability as health AI becomes more ubiquitous. It is important to understand that monitoring is only one part of the overall approach to equity, as considerations should be made to address bias and equity through appropriate choices in data sources and AI algorithmic design as well.

Lastly, the use of AI is iterative and cyclical in nature, mirroring the Learning Health System (LHS) by ingesting data created by health care processes to generate information and evidence that is fed back into the health care system. While potentially speeding up translation of data to knowledge to practice, this cyclical nature also means that without appropriate intervention, any biases and inequities present in the health care system and resulting data may be perpetuated and potentially magnified by AI applications (Leslie et al., 2021; Suresh and Guttag, 2021). It is important to conceptualize both AI research and downstream use as a virtuous cycle with each activity informing improvements and new directions in the other.

Commitment 3: Engage Impacted Individuals

Efforts in generating evidence regarding new frameworks, methods, and tools to support AI implementation in health care critically need representation from developers, subject-matter experts and end users, ethicists, patients, caregivers, organizational and cultural groups, and legal and policy makers. Intended as examples from the large numbers of stakeholders already noted, provided below are some specific engagement recommendations.

Researchers can engage federal agencies and non-profit foundations to advocate for increasing investment in foundational and applied health AI research that is in alignment with emerging best practices and for continued expansion of knowledge and understanding of all aspects of health AI from design through implementation and maintenance.

Researchers can actively engage with study participants and advocacy groups and communities that represent their interests—especially those who are underrepresented in research—in every stage of AI research, from conceptualization to design, execution, and interpretation and findings. This can include directly embedding patient and community representatives in the conduct of research but also promoting and advocating for patients and communities to be involved in research agenda setting and priorities. Additional research should be done in the

science of patient and community engagement to facilitate deeper integration throughout the AI lifecycle and in ways that ensure patient and community perspectives are integral to the process.

Researchers can engage health systems and payors to understand the business and clinical needs facing these organizations, and partner with them in the conduct of research to characterize, evaluate, and address challenges and gaps.

Commitment 4: Improve Workforce Well-Being

Research funding is needed to pursue activities that apply AI to reduce administrative burden, to improve human–human connection and communication, and to reduce clinical cognitive burden as well as to improve data quality, completeness, and timeliness. Research is needed to expand the understanding of the sociotechnical aspects of health AI as well as the best practices and standards in workflow integration. Numerous sociotechnical frameworks exist, but efforts to harmonize these learnings and promote maturity in this domain are important (Reddy et al., 2021; Salwei and Carayon, 2022). Researchers have an opportunity to play a leadership role in the development and validation of AI technologies, such as ambient scribes, that can reduce administrative burden, improve workforce-patient communication time, and reduce cognitive workforce burden.

Commitment 5: Monitor Performance

Researchers are poised to make major contributions, in partnership with other stakeholders noted, toward the development and promotion of methods and tools to facilitate the monitoring and sustainment of AI technologies’ performance in the context of use. AI algorithms should strive for explainability and plausibility at the model level, with clear assessments of data inputs and performance outputs being regularly conducted for efficacy, bias, equity, and safety to ensure trustworthiness of AI use (Adam et al., 2020; Feng et al., 2022). Researchers can also play a central role in expanding the understanding and capacity to anticipate unintended consequences, to establish best practices to prevent their occurrence, and to develop monitoring capacities to identify and mitigate them when they do occur (Suresh and Guttag, 2021).

Commitment 6: Innovate and Learn

Health care is a large-scale, dynamic system—and the “butterfly” effect of AI interventions can be significant and difficult to anticipate. AI’s speed and scope are

accelerating, sometimes at odds with the methodological, bias-controlled approach to research (Lorenz, 1972). AI systems are dynamic, more akin to behaviors than fixed output. Enhanced methods to dynamically monitor AI output and up-front checks on model behavior are required to ensure researchers’ ability to fulfill the principles of the ethical frameworks, especially in protecting the rights of and avoiding harm to people.

Sharing research about AI can be challenging, but researchers could develop, promote, and adopt reporting frameworks that facilitate transparency and reusability of AI research. Adaptations of frameworks such as FAIR and CARE (Carroll et al., 2021) and protocols for research such as Consolidated Standards of Reporting Trials for AI and Protocol for Development of a Reporting Guideline for AI are warranted to ensure optimal openness of research while considering the impacts on communities, especially those who have faced historical harms from research. These changes enhance the focus on fairness, separation of training and test sets, and suggest new ways to engage affected communities so they can exert more control and shape ethical approaches. Ultimately, algorithmic transparency and the research used to demonstrate its effectiveness will be crucial to achieve trust and enable AI sustainability. And given that private companies may be reticent to share their findings without a strong business case to do so, public funding of AI research will remain essential to developing trust.

In conclusion, the rapidly changing, large-scale nature of AI coupled with the complexity of health care can lead to unforeseen consequences and new requirements of researchers. Researchers have new responsibilities in ensuring the validity and equity of AI applications and implementation, monitoring and oversight of deployed AI systems in health care settings where they may exhibit emergent or unintended behaviors. Equity may be especially affected, as public perception of algorithms’ fairness may clash with the reality of algorithmic bias. Ultimately, humans, including researchers, are responsible and accountable for any biases and errors made by AI systems.

HEALTH SYSTEMS’ AND PAYORS’ PERSPECTIVE

As noted in the introduction of this chapter, the health system is experiencing significant challenges for which they are actively seeking AI solutions. Simultaneously there is a dearth of experience with these new tools and organizational-level governance of them. Yet, these organizations have a leadership role in specifying and promoting the business and clinical needs and requirements for the use of health AI and bear a strong responsibility for ensuring that AI used in care delivery benefits patients equitably and is used in a way that

builds trust in the health care system. This includes carefully considering issues including patient privacy, consent, agency, and accountability, addressing legal and financial responsibilities, and overcoming challenges related to the workforce. By implementing strategies designed to support these priorities, health systems and payors demonstrate accountability in their pledge to harness AI’s potential effectively, safeguard human connections, mitigate risks, and foster a more inclusive and trustworthy health care environment (Dorr et al., 2023). Outlined below are health systems’ and payors’ actions that are aligned with and advance the AICC principles and commitments, as well as the AI lifecycle where appropriate.

Commitment 1: Advance Humanity

While health systems and payors typically have patient health as a primary aim in their mission statement, in the context of health AI, they could go further to establish clear aims prioritizing patient health as paramount, and engaging patients and health care professionals as partners to identify key opportunities where AI can enhance human connection within health care settings. Health systems and payors can require that AI technologies that are to be used be adapted to the “local” context of use, be patient-centric, and incorporate appropriate local clinical and technical expertise and patient insights to ensure relevance and effectiveness (Sauerbrei et al., 2023).

Commitment 2: Ensure Equity

It is important that health systems and payors require transparent conduct and reporting of equity impact assessments of health AI to identify and mitigate potential biases or disparities in access and outcomes (Kim et al., 2024). These activities should involve patients and communities in the designing, developing, evaluating, and governing of AI applications, respecting their preferences and values, to ensure that AI technologies are patient-centered and benefit all parties equitably.

One of the largest challenges to ensuring equity in the use of health AI is the development of frameworks, processes, tools, and workflows that are scalable, inexpensive to implement and maintain, and are widely available to health consumers, particularly those in under-resourced environments or settings. The absence of these characteristics will promote inequitable access to health AI and restrict access of these technologies to only large, well-resourced systems. Health systems and payors can promote equity in this axis by selecting AI technologies that express scalable characteristics that make use and implementation easier.

It is important to recognize the historical and systemic sources of mistrust, harm, and bias that health system and payor behaviors have engendered (Brown et al., 2024; CDC, 2022). This stakeholder group should facilitate processes to ensure that AI does not exacerbate or perpetuate these issues by actively working to eliminate actions that erode trust or result in biases or harms within AI systems. One aspect of this could be requirements that health AI selected for use include patient-understandable, culturally appropriate information so that patients and their caregivers can assess AI’s potential risks and benefits in context with their care goals. This includes transparency regarding both the AI technologies and the patients’ health data used by and generated from such tools. Enhancing privacy (Murdoch, 2021) and consent (Perni et al., 2023) practices is also central to upholding equity and trust in AI applications and requires a multifaceted approach to promote transparency, adaptability, patient involvement, and regulation adherence. This includes establishing robust mechanisms to maintain accountability for, disclose occurrences of, and mitigate AI-based harms to uphold ethical standards and maintain patient trust, and regular review and updating of these policies in response to evolving technology and assessment of performance in the local setting.

Health systems and payors can provide a convening and facilitating role in promoting transparency and balance in the goals and values between health care delivery and patient goals and needs. It is important to establish a multifaceted approach to balancing goals and values among health and health care stakeholders that acknowledges differing values and goals and emphasizes shared benefits. Increasing transparency and accountability for health systems and payors, and information accessibility for patients, is paramount to support these processes.

It is important that health care organizations develop a culture of equity that includes making health equity a strategic priority, creating structures and processes to support equity work, and eliminating structural racism. These initiatives can help ensure equal distribution of benefit and risk for all (Wyatt et al., 2016). This includes aligning the incentives and payment models of patients and health systems with the objectives of improving health outcomes, quality, and equity for patients with a goal of mitigating the potential for AI misuse or financial exploitation.

Data governance, model development, and model implementation practices should promote equity by collecting diverse and representative datasets to ensure the AI system works effectively for all patient groups that the model seeks to benefit (Juhn et al., 2022). This includes regular performance reviews, which are important to identify and address emerging disparities, ensuring that benefits and risks are equitably distributed. In addition, organizations should

consider establishing robust mechanisms to maintain accountability for, disclose occurrences of, and mitigate AI-based harms to uphold ethical standards and maintain patient trust.

Lastly, as part of the overall efforts to maintain patient trust and deliver ethical care, it is important to adhere to regulations, standards, and certification for data quality, security, and governance, such as the Health Insurance Portability and Accountability Act (HIPAA) (HHS, 1996); the International Organization for Standardization (ISO) (ISO, n.d.); Health Data, Technology, and Interoperability (HTI-1) (ONC, 2024b); and The Joint Commission’s Responsible Use of Health Data certification (The Joint Commission, n.d.), as well as local governance standards and requirements.

Commitment 3: Engage Impacted Individuals

Health systems and payors should consider prioritizing the engagement of diverse patient groups, health care professionals, and other stakeholders throughout the AI lifecycle to ensure an AI application appropriately addresses patients’ needs and respects their values and preferences. Inclusion of ethics and equity experts can help guide the development process, ensuring that an AI solution’s primary aim is to advance human health. Continuous feedback from patients, clinicians, and other stakeholders should be gathered and used for ongoing improvements. This feedback should also be shared with patients, health systems, regulators, and accreditors to ensure accountability.

Health systems and payors can engage with developers to translate design requirements, goals, and targets into solutions that are adaptable, scalable, and capable of delivering tangible benefits. Partnering with developers and leveraging insights from federal agencies can ensure that AI models are technically proficient, ethically sound, and clinically relevant.

This stakeholder group should engage workforce and clinician well-being experts and researchers to collaborate in assessing the impact of AI solutions on clinical workflows and practitioner productivity and morale. In addition, they should engage AI users to determine their preferences for how and where the outputs of the AI solutions should be displayed and shared and promote collaboration and alignment with internal technology teams for optimal integration into workflows. Lastly, and most importantly, patients should be engaged to identify patient preferences in how, why, and when health AI should be utilized on their behalf, with a focus on goals of care and AI’s impact on any decisions made. AI solutions should prioritize patient health, privacy, and well-being (The Light Collective, 2024).

Commitment 4: Improve Workforce Well-Being

As employers of the health care workforce, the health systems and payors stakeholder group have a central role in sustaining and improving the workforce care delivery experience. It is important to prioritize AI technologies that are likely to significantly positively impact the health care delivery experience for the workforce, while also contributing to improved outcomes of care.

Health systems and payors should consider efforts to foster a culture of collaboration and shared purpose among the health care workforce and involve them with agency throughout the AI lifecycle. This includes involving them in decision making that influences the design and development or procurement of AI systems as well as use of regular feedback mechanisms, such as surveys and listening sessions. This should also include prioritizing human-centered AI design and careful workflow integration that fosters communication skills and capacity to preserve and enhance the quality of interactions between health care workers and patients. Organizations should also provide mechanisms and ongoing support for health care workers that includes accepting feedback and addressing moral and ethical concerns as humans learn to adapt and work effectively with AI systems. Effectively establishing this type of culture and transparent processes can facilitate the health care workforce in developing a sense of ownership, investment, and professional satisfaction in the outcomes of AI use.

Workforce training and support in the use of AI in health care delivery is essential for addressing concerns and fostering a positive workplace culture of collaboration and learning, promoting effective and ethical utilization of these technologies. This requires integrating AI training into continuing education offerings, promoting interdisciplinary learning, and establishing continuous professional development programs. To combat automation bias, it is important to enhance AI literacy through targeted and ongoing training on the strengths and limitations of AI. Consideration of decision support systems designed to augment human judgment and highlight potential automation bias for mitigation may be appropriate, as could workflow processes that require explicit review of AI outputs (“AI Timeouts”).

Health systems should consider adopting innovative collaboration strategies with health care workforce stakeholders to proactively address the potential displacement of human health care workers and the potential for degradation of human connections due to AI integration (Davenport and Kalakota, 2019). Possible strategies include engaging in open dialogues to discover needs and concerns, conducting workforce impact assessments, and promoting skill diversification toward roles where human expertise remains indispensable.

Commitment 5: Monitor Performance

For AI to be successful in health and health care, its output must be timely and integrated with dedicated infrastructure, resources, and trained personnel. Additionally, AI systems should adapt to changes in the health and health care landscape and focus on meaningful patient outcomes (Kwong et al., 2024), including leveraging higher-quality, real-world data (Silcox et al., 2024).

The efficacy and safety of tools may be compromised over time as evolving clinical environments disrupt the performance of the underlying models (Subbaswamy and Saria, 2020). To maintain the safety and effectiveness of AI algorithms in health care organizations, institutions should establish continuous learning cycles (Embí, 2021), providing ongoing performance monitoring, communication updates to users, and models adjustments or decommissioning as needed. This requires “close collaboration between clinicians, hospital administrators, information technology (IT) professionals, biostatisticians, model developers, and regulatory agencies” (Feng et al., 2022, p. 1). This stakeholder group should engage with appropriate stakeholder groups to establish processes for monitoring ongoing data quality and relevance, which includes transparent reporting of AI performance, including adverse events and success stories (Davis et al., 2020). This also includes the need to evaluate and monitor over-reliance on AI tools in some contexts (automation bias). Overall, promulgating transparency builds trust, ensures accountability, and promotes patient safety and public trust (van Genderen et al., 2024) in AI technologies.

Health systems and payors should consider developing transparent and comprehensive documentation (Brereton et al., 2023) and reporting, outlining how the AI system’s performance and impact on health and safety will be monitored and shared. This stakeholder group also plays a central role in promoting accountability, disclosure, transparency, and mitigation of AI-based harms’ legal and financial responsibilities in a comprehensive, patient-centered approach. This group also has a responsibility to share the outcomes and impacts of AI applications with other stakeholders, such as vendors, developers, and users, and coordination and facilitating clarity of these shared responsibilities is important for all stakeholders to acknowledge and act on.

Commitment 6: Innovate and Learn

This stakeholder group plays a coordinating role in implementation and maintenance of AI used within health care delivery and should consider adopting a forward-thinking approach to innovation and embrace continuous

learning and improvement based on emerging evidence and stakeholder feedback.

Health systems and payors could consider adopting transparent, strategic, and operational alignment of conduct, accountabilities, and relationships (Tabassi, 2023) throughout the AI lifecycle. This alignment is crucial for overcoming siloed approaches and translating ethical principles into actionable practices that drive AI adoption (Adler-Milstein et al., 2022) as a sociotechnical system (McCradden et al., 2023). In addition, defining organizational roles and responsibilities enhances accountability and facilitates effective change management. Stakeholders collectively enhance patient outcomes and operational efficiency through the accountable development, implementation, quality, and risk management of AI in health care (Overgaard et al., 2023).

As health systems and payors collaborate with developers to foster innovation and adapt to the regulatory landscape, it will be important to clarify expectations regarding testing and evaluation before implementing AI into clinical practice could accelerate the benefits experienced by patients. Collectively, stakeholders across the AI lifecycle ensure that AI solutions are ethically developed, effectively implemented, and continuously monitored to enhance patient care and maintain accountability (Bedoya et al., 2022; Raji et al., 2020).

To achieve the goals stated above, this stakeholder group can establish systems and local governance processes for ongoing evaluation, transparency, and communication regarding AI’s real-world performance and impact. Organizations can also adopt external standards and develop local standards for what constitutes adequate assessment and learning to know that AI innovations are safe, effective, efficient, patient-centered, and equitable. This ensures continuous improvement and advancement of clinical practice standards (Sendak et al., 2023).

In summary, by considering the AICC Commitments and the AI lifecycle in approaching and using health AI technologies, health systems and payors have an opportunity to optimize clinical outcomes, improve operational efficacy, and foster trust in the health care system. The strategies referenced provide a call to action to develop, implement, and monitor AI solutions in a manner that prioritizes transparency, accountability, and equity. However, realizing these goals requires concerted action from all involved parties.

PATIENTS’, FAMILIES’, AND THEIR ADVOCATES’ PERSPECTIVES

There is a growing awareness that the patient stakeholder voice has been under-represented in health AI (Adus et al., 2023). A recent systematic review of

diagnostic AI studies found that patient perspectives were half as likely as clinician perspectives to be represented, although the study was among English-only studies and represents a high-income country bias (Hogg et al., 2023; Kuo et al., 2024). Health AI should be designed with patients recognized as primary stakeholders, end users of AI, and co-creators of responsible AI. It is critical that the patient’s voice, needs, and interests shape the future of health AI, in collaboration with other AI ecosystem stakeholders, and patient-led governance and rights are essential for fair, ethical, and equitable integration of health AI in health care. Thus, below are considerations for actions that, as empowered stakeholders, patients and patient advocates could take to contribute to amplify the patient voice in health care AI and embed the Code Commitments into the ecosystem, with alignment to the AI where appropriate.

Commitment 1: Advance Humanity

Patient rights, interests, and unmet needs should be prioritized in the ideation, design, development, deployment, and governance of health AI solutions throughout the AI lifecycle (The Light Collective, 2024). Patients as well as patient advocacy and representation groups can actively engage with other key AI stakeholder groups and continue to advocate for patient stakeholder inclusion and promotion of patient agency throughout all aspects of AI ideation, design, development, implementation, monitoring, and improvement.

Commitment 2: Ensure Equity

Increasingly, institutions, clinicians, government agencies, and the health technology industry involved in developing health AI solutions are recognizing that diverse and under-represented patients and communities bear the greatest burden of risk for health inequities, bias, harm, and discrimination. It is important to advocate for a novel, necessary role in AI governance for patient-led representation, rooted in lived experience and expertise, and relevant to the context of intended AI use.

There are several distinct aspects to such advocacy. Patients and advocacy groups can assess the degrees of transparency and patient engagement among organizations using health AI, and advocate for increasing patient representation and equitable use of AI among other AI stakeholders as needed. This stakeholder group can also champion increased access to their data for health AI purposes within the context of the cultural values of that group, and the intended uses of the data, with appropriate protections to support expanded AI use that is equitable and representative of that

population. In addition to data access, transparency in and effective communication about what and how patient data are used in the AI lifecycle for all applications is essential for trust and requires ongoing attention by patients and advocates. It is also important to advocate for transparency of and access to the outputs of AI tools used in a patient’s clinical care. Lastly, this group can advocate and lobby for legal rights and protections for people and communities negatively impacted by health AI.

Commitment 3: Engage Impacted Individuals

There is an established science to patient engagement partnerships, which can be applied toward fulfilling this Commitment in the context of the AI lifecycle (NAM, 2022b; PCORI, n.d.). AI, by design, should mean co-creation with the patient voice as integral throughout (Vanstone et al., 2023). In the context of health AI, this includes promotion of mechanisms, methods, outreach, and frameworks to lower barriers to engage patients in each stage of the AI lifecycle. Patients and advocates might also work to promote awareness of patient sub-populations (e.g., those with specific diseases or cultural values and experiences) whose perspectives might be particularly suited to specific AI solutions.

Patients and advocacy groups have an opportunity to support the expansion of funding and investments in research that requires their engagement. They can advocate for research funding announcements to require patient participation in health AI research project design and oversight, and they can engage professional societies and present at research conferences to promote inclusion of patient perspectives at AI-related venues.

The advancement of explicit inclusion and representation for patients and advocacy groups in the governance and processes of AI use among health systems and payors is essential. Establishing standing patient representation groups collaborating with one or more organizations along specific patient missions related to disease management or cultural values can facilitate continued engagement.

Commitment 4: Improve Workforce Well-Being

As the health care workforce’s mission is to care for patients, this stakeholder group can play a central role in partnering with health care workers to improve the quality, safety, human connection, and trust in the delivery of health care. By advocating for health AI that meets patient needs, perspectives, and cultural values while simultaneously promoting outcomes that improve the workforce’s sense of purpose and capacity for patient engagement, patients and advocates may gain a greater sense of partnership with their clinical teams. For example, this group

could advocate for AI that improves the time health care workers can spend in direct interaction, as the most common reason for patients to lose trust in their provider was spending too little time with them (Birkhäuer, 2017; NORC, 2021).

Commitment 5: Monitor Performance

Trust in health AI will be aided by transparency in monitoring of AI technologies’ performance related to safety, efficacy, and equity that is comprehensible to patient communities and advocates. Patients and advocates could consider calling for concepts, frameworks, methods, and tools that can translate the technical and clinical specifications and needs of health AI into patient-centered, culturally appropriate language and interpretation. More specifically, examples could be to advocate for disclosures in accessible terminology similar to nutrition labels on foods (Sarasohn-Kahn, 2021), transparency of bias assessments, applicability of AI tools to the populations they are being used in, and safety and adverse event reporting in the context of use. If AI-generated clinical decision support outputs are used to make decisions about an individual’s care or coordination of care, this group can advocate for individuals having a protected right of access to these outputs, and request that said outputs be documented in the individual’s patient record. They can also advocate for such outputs being recognized as a part of a patient’s designated record set, recognized as electronic health information (EHI) (HHS, 2024c), and for having information blocking rules (ASTP ONC, 2024) applied (The Light Collective, 2024).

Commitment 6: Innovate and Learn

Given the novel issues that AI presents, particularly for patients who provide the data to train AI models (monetized by others) and who are recipients of AI outputs, sometimes without their explicit knowledge, innovation in governance is an important consideration to ensure that the primary aim of advancing humanity through health AI is achieved.

Patients and advocates can champion a requirement for patient representation that holds legally enforceable an independent duty of loyalty to continuously improve outcomes for patients. They can advance the imperative that this oversight have real authority, especially when industry or health system interests conflict with the public good, patient safety, and privacy.

This group can advocate that resources and funding be prioritized and allocated for diverse patient communities to conduct outreach education to build capacity for patient representatives to be informed consumers of AI and

facilitate capacity for patient organizations to establish and promote independent oversight. This investment could reinforce the concept that patient voices are integral to the development and deployment of health AI technologies in a sustainable and longitudinal manner.

Due to the transformative nature of AI, some aspects of individual consent may no longer be adequate to protect the health, safety, and well-being of patients. This group could call for stakeholders to proactively co-develop consent and community governance frameworks for health AI use.

It is important for patients and their care partners to understand what AI-powered clinical decision support (CDS) tools and innovations that different health care delivery systems and providers use. In the same way that patients may search a health system’s directory of physicians to explore their biographies and clinical interests, it would be useful for patients to be able to access a directory of CDS tools that may be utilized at a given health system. Patients may also want to see which physicians may incorporate these tools into their care. This level of openness can build trust and may invite shared decision making. It may also create opportunities to advance patient agency.

Advocacy could also include the right to opt out of the use of AI-powered technologies, especially if it increases the costs of their care without a significant impact on outcomes, the disclosure of what data were used to train specific AI-powered CDS tools that may be used in their care, and how their data are being used in relation to AI.

The American Hospital Association’s (AHA’s) Patient Care Partnership (formerly the Patient Bill of Rights) (AHA, 2003) provides an overview of what patients should expect during their hospital stay. Patients and patient advocacy groups could advocate for an update to this document to reflect new inclusions that relate to AI. This might include informing patients about the role of AI in their care if AI-powered CDS tools are used to guide diagnosis and treatment, as well as the potential benefits, risks, and limitations.

FEDERAL AGENCIES’ PERSPECTIVE

In 2024, the Department of Health and Human Services (HHS) reorganized technology, cybersecurity, data, AI, and policy functions under the Assistant Secretary for Technology Policy, Office of the National Coordinator for Health Information Technology (ASTP ONC) and has appointed a chief AI officer and a chief data officer to report to the chief technology officer (HHS, 2024b). And HHS published an AI Strategy, a Trusted AI Playbook, and its AI Strategic Plan (HHS, 2021, 2025, n.d.a). The Agency for Healthcare Research and Quality

(AHRQ) Digital Health Care Research Division reported in 2022 that 17% of its grants since 2020 involved health care AI research (AHRQ, 2022). CDC is funding partnerships with private-sector experts to develop a framework for identifying and preventing biases in public health uses of AI tools (CDC, 2022). More activity related to health AI is expected in the coming months in the wake of these changes.

Federal agencies could use these same authorities and tools to promote alignment with the AICC Commitments. This requires a recognition of the different statutory missions and authorities of each agency as well as a commitment to shared strategic and operational alignment to ensure a consistent message to developers, patients, caregivers and others who will be affected by the creation of new AI tools for health. Within that context, outlined below are considerations of ways in which federal agencies could incorporate the AICC Commitments in their internal research, development, and clinical care efforts, and in their external stakeholder engagement, standard setting, regulatory activities, and program priorities for funding. Where applicable, alignment of these activities with the AI are noted.

Commitment 1: Advance Humanity

This commitment directly aligns with HHS’s mission “to enhance the health and well-being of all Americans, by providing for effective health and human services and by fostering sound, sustained advances in the sciences underlying medicine, public health, and social services” (HHS, n.d.b). This concept is already central to all federal agencies involved in human health, including those outside of HHS. However, in the context of the challenges for health AI, federal agencies play a central role in providing appropriate protections, regulations, and policies to ensure continued human agency in health and health care.

• The National Science Foundation, CDC, AHRQ, U.S. Food and Drug Administration (FDA), the National Institutes of Health (NIH), the Health Resources and Services Administration (HRSA), and other federal agencies could fund additional studies to measure how AI can influence patient health, with particular emphasis on how AI influences human agency, goals of care, and human–human interactions in the presence of AI interventions.

Commitment 2: Ensure Equity

As noted above, federal agencies have existing policies, recommendations, and guidance to address this commitment. However, large gaps remain in effectively

ensuring equitable benefits and risks to all, across technical, sociotechnical, and cultural considerations and settings. This field is evolving rapidly, and agencies can play a pivotal role in shaping how bias and equity in health care are addressed through research investments as well as iterative policy and standards updating. ASTP ONC can support coordinated efforts to fill these gaps.

• NIH, the Centers for Medicare & Medicaid Services (CMS), CDC, and other agencies could fund additional research that uses AI to measure disparities; agencies could also further prioritize funding to evaluate the extent to which AI interventions reduce or increase disparities. All federal agencies that leverage AI prediction models could require representative sub-populations in models developed and used, as well as support expansion of methods and tools to comply with requirements for evaluation and mitigation of bias and inequitable performance in sub-populations.
• As assessment and mitigation of bias mature, FDA, ASTP ONC, and other regulatory agencies could iteratively expand requirements in the use of diverse and representative training and validation datasets to help improve performance across sub-populations.
• Federal agencies could continue to update data sharing requirements and develop and expand sustainable data sharing platforms such as HHS’s HealthData.gov to provide health AI application development with improved access to diverse and representative data.
• In undertaking enforcement, the ASTP ONC and the HHS Office for Civil Rights (OCR) could prioritize information blocking enforcement and HIPAA enforcement to enable individuals’ access to EHI, including EHI that is created by AI, to support openness and engagement of individuals. OCR has promulgated regulations (45 C.F.R. Part 92 implementing Section 1557 of the Affordable Care Act) that prohibit discrimination from patient care decision support tools. OCR could build on these regulations to support transparency regarding data sources.
• ASTP ONC could recognize standards for collection and exchange of relevant data and encourage use of the Trusted Exchange Framework and Common Agreement (TEFCA) for making data available for prioritized research projects, including using the individual access service. Adoption of the commitments could be required for researchers seeking access via TEFCA for such priority research data sharing (ASTP ONC, 2024).
• CMS could also enable compliance with the Code Commitments to protect and advance human health by prohibiting the use by health plans under its purview of AI-powered claims denial processes in ways that harm patients.

• CMS has already taken a significant step in this direction by requiring Medicare Advantage plans to make coverage decisions based on individual circumstances versus relying on generalized algorithms.
• OCR could expand guidance on how AI tools can be leveraged, consistent with HIPAA and applicable civil rights laws, such as Section 1557, and provide guidance on how to monitor for discriminatory impact of patient care decision support tools.

Commitment 3: Engage Impacted Individuals

• Research funding agencies could undertake and fund research and evaluation regarding stakeholder engagement in AI, evidenced by engagement records, diversity of input, and stakeholder satisfaction of research products.
• Federal agencies could expand stakeholder engagement to ensure that agency-wide funding opportunity announcements (FOAs), research funding decisions, regulatory approval processes and health care delivery standards include evolving concepts of inclusivity, equity, and representativeness in the context of use.

Commitment 4: Improve Workforce Well-Being

• CMS could monitor and fund others to assess the well-being of the health care workforce interacting with AI systems, using surveys, and assessing changes in job satisfaction and perceived workload.
• CDC could continue to expand evaluation of the state of the public health workforce through development, integration, and assessment of new AI tools that identify potential ways to enhance the workforce well-being.

Commitment 5: Monitor Performance

• NIH could collect and make public information on AI system performance, including access to performance data, adherence to regulatory reporting requirements, and active dissemination of findings to all stakeholders. This could be achieved through a system such as ClinicalTrials.gov, which is supported by the National Library of Medicine and FDA.
• ASTP ONC has already taken action to require transparency and risk management for AI-assisted decision support interventions that are part of certified EHR products (HHS, 2023a); ASTP ONC could evaluate and consider the Code Principles and Code Commitments for inclusion in regulations and guidance regarding conditions and maintenance of certification.

• The Veterans Health Administration (VHA) could promote this commitment through open sharing of data, algorithms, performance, and metrics of AI solutions implemented within VHA.
• The Cybersecurity and Infrastructure Security Agency could use its authorities and influence to promote additional transparency and standardized reporting requirements for security and privacy assessments of AI tools before, during, and after deployment.

Commitment 6: Innovate and Learn

As part of the overall ecosystem of safe and effective innovation and continuous improvement, federal agencies could incorporate the Code Principles and Code Commitments, where not already present, in FOAs for research (Public Health Service agencies such as NIH and CDC), criteria for product approvals (FDA), and conditions of participation in Medicare and Medicaid (CMS).

• NIH, CMS, FDA, ASTP ONC, and other federal agencies could expand funding for research into the rate and impact of continuous improvements to AI systems, evidenced by update logs, the implementation of feedback into system refinements, and innovation metrics.
• HRSA, the Indian Health Service, VHA, and the Department of Defense could expand research into the development and implementation of AI-supported CDS tools integrated into health IT systems that address inclusion and equity.
• FDA has already issued an action plan for AI-based, medical device software (FDA, 2021) and put forth discussion papers on AI in drug development (FDA, n.d.). The agency could propose or require additional levels of adherence with the collection of real-world evidence that adheres to broad diversity and equity considerations (FDA, 2024).
• CMS has already established payment methodologies for reimbursement of AI products (Chen et al., 2021); the agency could further adopt reimbursement standards that prioritize compliance with the Code Commitments and products that are built on or implement the commitments.
• ASTP ONC could expand certification standards or conditions of certification in health IT technologies, including but not limited to EHRs, regarding continuous learning and updating AI over time.
• As the Health Systems & Payor workgroup observed, collaboration between researchers and federal agencies can set the standards for necessary oversight and support for adapting AI solutions to emerging challenges and shifting patient needs. HRSA and CMS both have authorities to lead on this work.

• FDA could expand requirements in AI product labeling based on real-world performance and other characteristics that would assist users and patients in understanding the context and performance of the use of the tools.

HEALTH AND HEALTH CARE WORKFORCE PERSPECTIVE

The well-being of the health and health care workforce has faced numerous challenges over the years from various occupational stressors, such as a historical culture of professional focus that challenged work/life balance, usability issues and time spent on the EHR, and most recently the COVID-19 pandemic (Alobayli et al., 2023; Shanafelt, 2021). In addition, well-being and global health care delivery are threatened by a growing shortage of health care personnel due to several factors, with the World Health Organization (WHO) estimating that by 2030 there will be a shortage of 14 million nurses, physicians, midwives, and other health care professionals (de Vries et al., 2023; WHO, 2016). In the setting of this growing crisis and amidst a health care culture adopting an increasingly proactive framework of systems-based interventions to identify and address causes of occupational distress (NAM, 2019), AI has substantial potential to harm or renew the well-being of the health care workforce. Presented below are key opportunities and challenges in the use of AI in health care in the context of the Commitments through the lens of Commitment 4: Improve Workforce Well-Being.

In addition, health systems and payors should leverage lessons learned from EHR implementation, which is discussed in another context in Chapter 2. EHR use, promoted by the Health Information Technology for Economic and Clinical Health (HITECH) Act, has been associated with decreased incidence of adverse events and occasionally improved coordination of team-based care and chronic disease management, but also decreased job satisfaction and higher burnout rates (Bates et al., 1998; O’Malley et al., 2015; Peccoralo et al., 2021; Reed et al., 2012). For EHRs deployed in environments with heavy regulatory compliance requirements and documentation burdens to satisfy multipayor reimbursement paradigms, perceptions that EHRs facilitate billing rather than clinical care are unsurprising (Holmgren et al., 2021; Tseng et al., 2018). Such challenges, which were anticipated by a 2009 National Research Council report (NRC, 2009), elucidate opportunities to leverage EHRs in reducing cognitive load and increasing cognitive support (Johnson and Stead, 2022). These opportunities are germane to AI tools in health care, which have the potential to reduce cognitive burden at the price of imparting automation bias in which health care workers experience a loss of critical thinking, judgment, and intuition—elements that must be preserved in AI-enabled decision support design and deployment (IOM, 2003).

Commitment 1: Advance Humanity

Integrating AI into health care workflows is a challenging task, and could involve functions such as automation, cognitive support, or information synthesis, among other opportunities. However, health care AI applications could degrade human connections among health care workers and between providers and their patients by automating tasks that have, historically, involved human-to-human interaction (e.g., a clinician obtaining sensitive information from a patient or their caregivers). Conversely, automating some basic tasks (e.g., data capture, medical coding) could reserve time and space for health care workers to be with one another and patients. Another challenge is that if simple, straightforward tasks are automated, humans could face workdays exclusively comprising complex, demanding tasks (e.g., a computer vision model diagnosing unremarkable and low-acuity chest radiographs while triaging images with moderate or severe pathology to human radiologists, who see nothing but disease, injury, and diagnostic challenges) (NEJM-AI Grand Rounds, 2023). Balancing the needs and preferences of all stakeholders while simultaneously prioritizing human health is essential to ensure that AI advances human agency and connection. User-centered and equitable approaches to the design and deployment of AI tools are best practices and are required to ensure they meet the varied needs of patients and health care providers effectively (Van den Bruel et al., 2010, 2012).

Commitment 2: Ensure Equity

The health workforce has a critical role in addressing disparities in health care service delivery, and workforce diversity is foundational for health equity (Pittman et al., 2021). Occupational stressors can be felt differentially among constituencies in the health care workforce, as was found in higher rates of unemployment among those self-identifying as Black or Indigenous following the COVID-19 pandemic (Semprini, 2023). AI technologies may be implemented in ways that are not culturally aligned with practice preferences for under-represented health care workers, or not aligned with patient preferences, and exacerbate rather than ameliorate workforce well-being and reduce equitable care (Parag et al., 2023). Care should be taken to understand the context of use for AI technologies and anticipate their use in under-represented populations and environments (e.g., rural practices and community health centers).

Commitment 3: Engage Impacted Individuals

Deploying AI in health care settings risks displacing health care workers, such as when AI replaces human work products (e.g., AI performing diagnostic tasks on radiographic and pathologic images). In the automotive industry, when robotic arms replaced assembly line workers for basic tasks, automobile prices fell while human workers experienced job loss. Eventually, jobs shifted toward tasks requiring creativity, long-term planning, and moral deliberation, skills that remain relevant for recent automotive industry endeavors, such as self-driving cars (Awad et al., 2018).

Given the high probability of similar job shifts in health care, it will be important to identify training and employment needs in areas less likely to experience AI automation, and training and retraining humans to work with AI. Education-enabled health care workforce job shifts, if successful, could not only mitigate risk of job loss, but also address global health care workforce shortages by maintaining current workforce volumes while addressing occupational stressors through workflow and work product optimizations with AI tools. For example, while not yet ready for widespread use, AI-enabled medical interpretation services may at some point replace human medical interpreters, easing a serious resource shortage and improving patient and provider experience (Lion et al., 2024); retraining those workers for roles that require more creativity, planning, and moral deliberation could benefit the workforce and the system.

Commitment 5: Monitor Performance

One of the challenges in health care workforce well-being has been the measurement and assessment of well-being in complex settings. Historically, most validated instruments of well-being assessment were manual and not easily monitored in an automated fashion (Boskma et al., 2023). The use of AI to predict well-being for downstream management is a growing field (Levin et al., 2024; Nan et al., 2024) with opportunities to identify policies, workflows, environments, and circumstances that negatively impact the ability to improve workforce wellbeing (Malgaroli et al., 2023). However, if these tools are not appropriately used, they could directly harm health care workers and lower well-being. As noted elsewhere, any AI tools put into place for these types of activities themselves should be monitored for performance and appropriateness in the context of use.

In addition, from a legal perspective, health care workers who use AI tools may incur personal liability for AI-associated errors or patient harm (Mello and Guha, 2024). As tort doctrine evolves to address the distinctive challenges posed

by AI, health care workers must be informed of the risk for personal liability and relevant institutional policies and should be protected by surveillance of AI-associated outcomes that adapts to the probability and severity of adverse events (i.e., higher-risk AI tools will require heavier surveillance). Those who purchase AI tools may have contracting opportunities to shift liability to developers when model outputs are erroneous and cause harm, while the purchaser or health care worker would remain liable for AI tool misuse (Banja et al., 2022). Payors could also protect health care workers and patients by insisting that AI tools are demonstrably safe, effective, and meeting salient (e.g., FDA Software as Medical Device) standards (Stern et al., 2022).

Commitment 6: Innovate and Learn

Effects of AI on the health care workforce will likely include elements that are currently unknown and cannot be foreseen. Mitigating risk for “unknown unknown” or “black swan” events should focus not on prediction—which is impossible—but rather should identify areas of vulnerability and build robustness against them (Rumsfeld, 2011; Taleb, 2007). LHS principles that support the translation of data to knowledge and to practice are applicable here. The goal is to ensure that clinical and operational knowledge gained from AI development and implementation is incorporated in AI applications that affect care delivery in a manner that compounds as the health of patients, health care workers, and health care system evolve (IOM, 2011). This approach may facilitate early detection and effective rethinking and adaptation when impactful, unforeseen events occur. Like all other strategies and tactics pertaining to health care workforce considerations, this approach must involve and represent all stakeholders in shifting the influence of AI away from harm and toward the well-being of health care workers.

Similarly, when some types of AI applications become reimbursable, there is risk for incentive-driven overuse and increasing documentation burdens for billing purposes. These risks may be mitigated via value-based incentives informed by audits of patient- and provider-level associations among AI use, coding practices, and patient outcomes.

HEALTH CARE QUALITY AND PATIENT SAFETY PERSPECTIVE

In 1998, the Institute of Medicine (IOM) Roundtable on Health Care Quality described an urgent need to improve the quality of health care, defined as “the degree to which health care services for individuals and populations increase

the likelihood of desired health outcomes and are consistent with current professional knowledge” (Chassin and Galvin, 1998). Harms related to overuse, underuse, and misuse were identified and further described in patient safety literature (Emanuel et al., 2008). Despite significant efforts, improvement and comprehensive measurement of health care quality and patient safety have proven challenging. In their article, “Two Decades Since To Err Is Human: An Assessment of Progress and Emerging Priorities in Patient Safety,” the authors noted that “the frequency of preventable harm remains high, and new scientific and policy approaches to address both prior and emerging risk areas are imperative” (Bates and Singh, 2018). Additionally, measurement is critical to improving quality, and while there is considerable dissatisfaction with the current state of measurement and assessment, few serious efforts exist to make significant changes (McGlynn, 2020).

There are “islands of excellence” as is demonstrated by the reduction in adverse events associated with major surgical procedures, cardiac events, and pneumonia between 2010 and 2019 (Eldridge et al., 2022). However, in 2022, the U.S. Office of Inspector General reported that 25% of Medicare patients experienced harm during their hospital stays during its study period (OIG HHS, 2022). Post-pandemic, U.S. health care safety declined precipitously and severely (Fleisher et al., 2022). Ensuring that the health care system heals rather than harms patients remains a critical priority for the United States; however, routine, comprehensive, and systematic national assessments of safety have not been established. In fact, the last national assessment was conducted in 2003 (McGlynn, 2020).

The Role of AI in Improving Quality

As described in previous sections of this chapter, AI has significant potential to enhance the quality of health care through improved prevention, early detection, diagnostics, and early intervention, treatments, and rehabilitation. AI may also be a contributing factor in catalyzing a new era of quality improvement through enhancing the accuracy, efficiency, and comprehensiveness of quality measurement, while making it substantially less burdensome. AI’s capabilities to process vast amounts of data rapidly and accurately contribute to enhanced quality measurement in health care (Jiang et al., 2017).

Quality measurement begins with data collection and integration. AI technologies can streamline data collection from various sources including EHRs, social determinants of health databases, wearable devices, and patient self-reports. By automating the extraction and integration of data, AI reduces the burden on health care professionals and ensures more comprehensive and accurate datasets (Esteva et al., 2019). This comprehensive data collection enables a more detailed and nuanced understanding of patient outcomes and care quality. Additionally, AI

has the ability to process and evaluate vast datasets, identifying relationships and repeated patterns that might otherwise go unnoticed by humans or be undetected using traditional statistical techniques (Shickel et al., 2018).

Traditional quality measurement often relies on standardized metrics that may not fully capture the unique needs and circumstances of individual patients. AI can create personalized quality metrics by including information about the individual, such as specific health conditions, treatment preferences, and social determinants of health (Obermeyer and Emanuel, 2016). Personalized metrics provide a more accurate reflection of care quality and patient satisfaction, in that AI can improve the measurement of patient outcomes by providing more sophisticated tools for evaluating treatment effectiveness and patient satisfaction. For instance, AI-driven sentiment analysis can assess patient feedback from surveys and social media, offering insights into patient experiences and areas for improvement (Topol, 2019). AI can also track long-term outcomes more effectively, ensuring that quality assessments reflect the sustained impact of care interventions. By analyzing data across multiple health care institutions, AI can identify best practices and facilitate benchmarking. Comparative analyses can highlight high-performing institutions and effective care strategies, guiding quality improvement initiatives (Rajkomar et al., 2019).

The Role of AI in Improving Patient Safety

Multiple interconnected factors influence patient safety. For example, harms can result from challenges with data, interoperability, system usability, clinician burden, and complexity of clinical scenarios (Tighe et al., 2024). AI, with its capacity to incorporate vast and disparate datasets, represents an important tool that holds vast potential to improve patient safety. AI has been successfully tested across eight major harm domains, and situations where new or unstructured data sources are incorporated to improve predictions are expected to yield the highest impact on patient safety. These novel data include sensing technologies such as “vital sign monitoring, wearables, pressure sensors, and computer vision” (Bates et al., 2021).

Using these data sources, AI models are poised to significantly improve risk prediction and thereby advance patient safety. Integrating data across novel and disparate sources in real time and adapting to continuously updated data, AI models can deliver timelier and more accurate predictions based on information such as biometric, sensor data, or video, sources that are currently untapped or abstruse (Tighe et al., 2024). An example of this is the use of movement trackers and camera data to identify fall risk (Tighe et al., 2024).

AI and Patient Safety and Quality in the Context of the Code Commitments

While the potential for improving health care quality and patient safety with AI is high, it also comes with opportunities and risks beyond those discussed previously. The Code Commitments offer additional guidance to mitigate these risks.

Commitment 1: Advance Humanity

As previously described, AI has the potential to significantly improve both human health and well-being. This must remain the litmus test for all applications of AI in health care. Generative AI can also create high-quality human-like responses to patient inquiries (Ayers et al., 2023);however, such responses—agency enhancing and available free online to individuals—can produce information that is inaccurate and potentially harmful (Coiera et al., 2023). It will be imperative to educate the public about the benefits and risks of AI that appears to provide empathetic medical advice outside the confines of trained health professionals.

Commitment 2: Ensure Equity

Equity is not limited to the outputs of AI but also to the access to the benefits of AI. If AI systems that improve quality and reduce harm are implemented, ensuring that those systems are available to low-resourced health systems will be essential for true equity. AI can be particularly susceptible to inequitable performance from biased training data and misaligned design but also can be designed explicitly to mitigate bias and promote equity (Cary et al., 2023) if conducted in a systematic way. Consideration of financial incentives for implementation of proven AI systems by small community and rural hospitals is warranted.

Commitment 3: Engage Impacted Individuals

The potential of AI to cause harm if poorly designed and implemented is as significant as with EHR implementation which contributed to an increase in errors and patient safety concerns due to information overload, presentation of irrelevant information, and data display issues (Nijor et al., 2022). Even more complex, when AI is integrated into EHR-based CDS tools, then potential safety challenges may occur from either or both in synergy and may still contribute to the already substantial problem of alert fatigue. Lack of user engagement in system design and implementation of new workflows can result in mistrust among patients and clinicians, diminishing the anticipated gain from the use of AI

(Tighe et al., 2024). Ensuring collaboration with clinical and administrative teams in the design and implementation of AI systems will be essential.

Commitment 4: Improve Workforce Well-Being

The application of AI in health care presents opportunities to reduce burden for clinicians; however, it also presents risks of over-reliance on AI, clinical deskilling, and ineffective human oversight. Relying on manual review of AI-generated decisions or recommendations is a weak approach to ensuring patient safety (Lucian Leape Institute, 2024). It is highly likely that as AI tools become more prevalent, clinicians will not be able to consistently review the output of AI scribes, bots, and other tools to spot errors or carefully reflect on the diagnostic or therapeutic suggestions of decision support tools. Thus, the error rate or risk of harm from any given AI tool needs to be considered on its own, not presented as if double-checking by clinicians will prevent those errors from impacting patients (Lucian Leape Institute, 2024).

Additionally, health care organizations, seeking to defray the costs of acquiring AI tools, could use AI-driven efficiencies to assign more duties to clinicians, nullifying anticipated improvements in workloads and cognitive burdens and, combined with de-skilling, further reduce professional satisfaction. AI will not produce the anticipated benefits in health care if the underlying health care payment model, which continues to pay for quantity over quality, is not simultaneously addressed.

A recent survey by the National Association for Healthcare Quality (NAHQ) noted that the health care quality and safety workforce often have a low level of competency in health data analytics and performance and process improvement (NAHQ, 2022). It is critical to facilitate and promote a quality and safety culture that has the training to adapt to an AI-enabled health environment.

Commitment 5: Monitor Performance

To proactively monitor and mitigate risks to patient safety, AI models and tools require continuous monitoring after implementation (Feng et al., 2022). AI models can be designed to monitor other AI models and tools to detect drift in the underlying data, performance loss over time, and differential performance in sub-populations (Davis et al., 2017). It may be advisable to consider the creation of a specific area of focus within health systems accountable for quality management and continuous improvement of AI models and tools, which have been referred to as “AI-QI” units (Feng et al., 2022).

Commitment 6: Innovate and Learn

Key to quality improvement and patient safety is seeking to continuously learn and improve care based on data and experience. AI offers immense opportunities to advance quality and safety in an LHS. Prioritizing research in this arena is essential to creating a strong evidence base for improving health outcomes. In addition, ensuring collaborative knowledge sharing across health systems will be important to rapid system-wide learning and diffusion of innovation.

AI is poised to support major advances in patient safety and quality measurement and improvement in health care. Efforts to leverage these opportunities must be balanced by those to mitigate the risks in an evolving landscape, aligning with the Code Principles and Code Commitments to ensure safe, high-quality, and trustworthy AI in an LHS.

BIOETHICS AND EQUITY PERSPECTIVES

The objective of this section is to highlight ethical and equity aspects related to the Code Principles and Code Commitments that may be in potential conflict as well as some examples for how to resolve such potential conflicts. Here, core principles and concepts from bioethics are brought together with attention to equity, conceptualized as the social and structural factors that affect individuals’ and populations’ abilities to achieve health.

There are a number of relevant ethical conceptual frameworks applied in scrutinizing ethics in AI (Heilinger, 2022; Zong and Matias, 2022). For this section, the Code Principles are conceptualized as substantive principles, meaning that they help guide how AI should be implemented in terms of characteristics that directly impact human health. The Code Commitments are viewed as procedural, meaning that they should help guide the process for how AI tools are developed, and help guide balancing and deliberations when substantive or other principles or values come into conflict.

From an ethics and equity perspective, it is important to go beyond the list of principles and commitments and consider how these may be in tension with one another; how—in case of tensions—they may be weighted or how this tension should be considered, negotiated, and even resolved; and even how they may complement each other. Detailed below are examples of such tensions to illustrate the importance of this perspective.

Safe and Effective May Come into Tension with Transparent

Emphasizing or requiring opening of the black box may reduce the safety or effectiveness of an AI tool in health. This may occur as a function of cognitive reliance on the AI tool without the ability to assess underlying information (Jabbour et al., 2023), which may result in reduced safety. It may also occur if understanding the data context or patient context supporting the AI could result in improved shared decision making (effectiveness).

Adaptive May Come into Tension with Accountable

Adaptive involves continuous learning and improvement, which are laudable goals, but ongoing change and evolution may make it challenging to maintain “clear accountability for potentially adverse consequences.” Put simply, change may cause ambiguity regarding who is accountable for AI-caused harms.

Equitable May Come into Tension with Secure

Prioritizing equitable access to data, for example, can compromise considerations related to protection of privacy. This may occur in the setting where patient anonymity cannot be assured for data contributed by under-represented patients (Brown et al., 2023). Inclusions of data from these patients increases privacy risk, not including data from these patients decreases equity in AI development and performance.

Engaged May Come into Tension with Efficient

Engaged involves prioritizing the needs, preferences, and goals of people, which may not always align with the goals of reducing costs for health gained or provide the most rapid mechanism to implement AI.

The Code Commitments may assist in identifying ways to resolve such tensions or recognize complementarity between principles. For example, “involvement” tells us to always engage people as partners. This means that some conflicts may be resolved by discussing with the relevant partners what matters to them in specific contexts. The use cases that follow provide examples of how the Code Principles may come into tension, and how turning to the Code Commitments as procedural tools may offer some paths forward.

Illustrative Scenario 1: Tension Between Safety and Transparency: Documentation Support

Clinicians often grapple with burnout associated with administrative burden of EHR documentation such as completion of clinical notes to ensure accurate billing and addressing inbox messages from patients, care team members, and consultants (Budd, 2023; Tran et al., 2019). Generative AI may be able to automate much of this administrative work and allow clinicians to focus on patient care (Reddy, 2024). Studies show AI-assisted documentation reduces burdens and frees clinicians to better serve patients (Tierney et al., 2024). AI has also shown the capacity to resolve inaccuracies in EHRs, such as discrepancies in medications, thus enhancing patient safety (Damiani et al., 2023).

However, the use of such AI tools raises concerns that patients may not understand how AI is being used in their care, potentially compromising their autonomy. Even if patients have the option to consent to—or decline—the use of AI, they may not fully comprehend what they are consenting to or declining. Moreover, if AI support in documentation is demonstrated to improve patient safety in certain use cases and becomes standard of care, equity challenges can arise if patients decline the use of these tools, especially if the decliners disproportionately come from marginalized populations. Hence, safety and transparency may come into conflict, raising ethics and equity issues.

One possible approach to such conflicts would be, as suggested above, to rely on the Code Commitments for guidance. Focusing on human health and connection, ensuring equitable distribution of benefits, involving people as partners, and promoting transparent stakeholder-inclusive prioritization between conflicts are all procedural requirements that can help negotiate such tensions in specific use cases.

Illustrative Scenario 2: Tension Between Adaptability and Accountability: Automated Diagnosis

IDx-DR was the first FDA-approved autonomous medical AI device. This tool diagnoses an eye disease, diabetic retinopathy (DR), by analyzing images of the eye. This AI device is significant because it provides a diagnosis, rather than advice to a clinician about a disease or condition (Van der Heijden et al., 2018). IDx-DR can provide a diagnosis and can be used by non-specialist clinicians, whereas previously this condition would have to be diagnosed by a specialist. Because of AI-assisted DR detection tools, DR can be diagnosed by a primary care physician. There is already promising evidence that these tools are allowing more people to

be screened for this disease, which can lead to improved health outcomes, such as increases in patients receiving eye screening exams (Knapp et al., 2023). There are increasing numbers of studies evaluating the efficacy and utility of automated DR detection (ADRD) (Joseph et al., 2024).

ADRD systems are powered by machine learning, which means that their diagnostic capabilities are derived from training data. In this way, these are adaptive tools, which is in line with the Code Principle of adaptability. Although adaptability is important for its primary diagnostic function, this adaptability can be in tension with the principle of accountability. Clinician users and patients may not be aware of changes in performance or limitations in environment or setting, and it might be difficult for clinicians and developers to account for or explain the source of errors. In this case, when adaptability and accountability can be in conflict, it is important to turn to the Code Commitments to balance proposals for actions and next steps.

In this case, both the adaptability and the accountability of the ADRD tools advance human health. In these situations, prioritization of one commitment over another needs to be done, and this should be conducted in the context of use, with engagement from all relevant stakeholders. In the setting of this section focused on ethics and equity, a prioritization of accountability promotes equity and prompts reflection on whether the errors might be disproportionately occurring within certain populations. There are other situations in which an AI tool is used in life-threatening clinical situations in which highly dynamic prespecified change control policies need to be in place to ensure safe operation. In this case, adaptability of the tool for safety reasons may outweigh accountability considerations.

Illustrative Scenario 3: Tension Between Security and Equity: Secondary Use of Data and Consent

There has been a long-standing tension in the secondary use of routinely collected health data between opt-in and opt-out consent procedures (de Man et al., 2023; Sanderson et al., 2017). While there have been variations in individual studies, a majority of patients interviewed or surveyed expressed both a desire for control over their data and a willingness to participate in data sharing for promotion of human health and particularly the health conditions of interest to the patient, and with less interest in those applications with commercially profitable objectives (Kalkman et al., 2022; Mikkelsen et al., 2023; Skovgaard et al., 2019). In health AI, this can be challenging as these goals are pursued simultaneously in many applications.

Perhaps counterintuitively, opt-in models—which appear to improve security and privacy and individual control of health data—may result in bias in consented datasets. A recent meta-analysis of opt-in versus opt-out studies found that the average weighted consent rate was 84% for opt-in and 96.8% for opt-out; however, when both procedures were explained “the consent rate was 21% in the opt-in group and 95.6% in the opt-out group” (de Man et al., 2023). In the context of data use for health AI development, this becomes particularly challenging due to the volume of data necessary to rigorously train these algorithms. Additionally, across the studies, the opt-in models yielded more biased datasets, with represented individuals more likely to be male, have more education, higher earnings, and improved overall socioeconomic position (de Man et al., 2023). While bias in health AI can come from several sources, bias in source data used for training is one of the most established and studied (Ferrara, 2023). Among these, selection bias can result in lower performance and disadvantage among under-represented populations in the data (Haneuse, 2016; Johnson et al., 2000).

In this case, the security and equity associated with secondary data use can appear to be in conflict, but the commitment to ensure equitable risks and benefits prompts reflection about the benefits of secondary use of health data for AI being disproportionately provided to some sub-populations and withheld from others. Additionally, the commitment to transparently monitor performance, including for evidence of bias, both proactively and retrospectively, can reduce the risk of disproportionate benefits or risks to any subgroup.

Considering the Code Principles and Code Commitments together to address apparent tensions is a valuable construct for researchers, developers, patients, and other stakeholders who can use them as a guidepost for designing, developing, implementing, monitoring, and maintaining AI tools in health care.

COMMON THEMES AND CONTRIBUTIONS ACROSS STAKEHOLDERS’ PERSPECTIVES

The expert working groups, representing broad stakeholder constituency, considered important actions to disaggregate and advance the Code Commitments in the context of the AI lifecycle. Both similarities and differences were identified between and among the outputs of the working groups. Summarizing these commonalities and distinctions, Table 5-2 identifies needed actions repeatedly identified across groups, while Table 5-3 describes the role each stakeholder group can play in applying the Code Principles and Code Commitments.

TABLE 5-2 | Common Themes for Action Among Expert Working Groups

TABLE 5-3 | Distinct Contributions of Various Stakeholders to the Application of the AI Code of Conduct Framework

This page intentionally left blank.