Naval Studies Board

Testing, Evaluating, and Assessing Artificial Intelligence-Enabled Systems under Operational Conditions for the Department of the Air Force Kickoff Meeting #1, Part 1

April 22, 2022

3:00 PM - 5:00 PM (EDT)

Past

Agenda

Download the Agenda

About

A kickoff meeting for the AFSB study on Testing, Evaluating, and Assessing Artificial Intelligence-Enabled Systems under Operational Conditions for the Department of the Air Force. We will hold the required project start-up briefings and discussions, hear from the project sponsor, and begin to plan the rest of the study.

Additional Information

Is it a Closed Session event?

Partially Closed

The following committee members were present at the closed sessions of the event

May Casterline
Tom Longstaff
Rama Chellappa
Jack Shanahan
Bob Bond
Rebecca Willett
Robin Murphy
David Rosenblum
Tamara Kolda
Craig Baker
Melvin Greer
Trevor Darrell
Humberto Silva III
Nandi O Leslie

The following topics were discussed in the closed sessions:

The required conflict and bias discussion for the project was held led by Scott Weidman, DEPS Deputy Executive Director

Date of posting of Closed Session Summary:

April 22, 2022

Organizing Project

Testing, Evaluating, and Assessing Artificial Intelligence-Enabled Systems under Operational Conditions for the Department of the Air Force

Contact

Ryan Murphy

rmurphy@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Meeting Closed

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #7

April 12 - 14, 2022

Past

About

This is meeting seven of the Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure. The meeting will be held at The Mark Center, for briefings. Returning to the Keck Center to continue writing/reviewing the report.

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Kevin Bowcutt, Katherine Graef, Conrad Grand, Michael Hamel, Heidi Perry, Terry Lewis, Al Sciarretta

The following topics were discussed in the closed sessions:

Briefings from TRMC, TSMU, and INOPACCOM

Date of posting of Closed Session Summary:

January 19, 2023

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Meeting Closed

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #6

March 17 - 18, 2022

Past

About

A committee meeting featuring classified data gathering sessions from DARPA; Director of Test and Evaluation, U.S. Air Force; Airborne Electronic Attack (AEA) Systems Program Office, NAVAIR; and Director, Operational Test and Evaluation (DOT&E), OSD. Additionally, day two will involve committee writing sessions. This meeting is not open to the public.

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Perry, Heidi
Bowcutt, Kevin
Graef, Katherine J.
Grant, Conrad
Greer, Ed
Hamel, Michael
Jackson, Dana "Keoki"
Lewis, Terry P.
Sciarretta, Al

The following topics were discussed in the closed sessions:

Discussion of workshop; findings, conclusions, recommendations; modelling and simulation in test and evaluation; debrief of White Sands Missile Range.

The following materials (written documents) were made available to the committee in the closed sessions:

All documents are classified or CUI, and unable to be shared with the public.

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Workshop Closed

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Workshop

March 16, 2022

8:00 AM - 5:00 PM (EDT)

Past

About

This classified workshop explores the unique challenges of testing emerging technology in the space domain, and what a space range may look like.

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Perry, Heidi
Bowcutt, Kevin
Graef, Katherine J.
Grant, Conrad
Greer, Ed
Hamel, Michael
Jackson, Dana "Keoki"
Lewis, Terry P.
Sciarretta, Al

The following topics were discussed in the closed sessions:

Report status, findings, conclusions, and recommendations, and briefings from Space Force Deltas and National Labs.

The following materials (written documents) were made available to the committee in the closed sessions:

All materials presented were classified.

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Assessment of Partnership Options for a Small Satellite System for Collecting Scientific Quality Oceanic and Coastal Data - Report Dissemination

March 3, 2022

2:30 PM - 3:30 PM (EST)

Past

About

This will be a webinar to disseminate the report - Leveraging Commercial Space for Earth and Ocean Remote Sensing

For Q&A, please use the following link: http://nationalacademies.org/deps-questions

Additional Information

Is it a Closed Session event?

Not Closed

Organizing Project

Assessment of Partnership Options for a Small Satellite System for Collecting Scientific Quality Oceanic and Coastal Data

Contact

Joe Czika

jczika@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #5

February 23 - 25, 2022

Past

About

Committee members will engage in data-gathering sessions with DARPA, the Army Test and Evaluation Command (ATEC), and Army and Navy T&E Executives.

Committee members will also discuss report writing, past and future site visits, and findings/conclusions/recommendations for the report. On Friday, the committee will draft sections of the report.

Additional Information

Is it a Closed Session event?

Partially Closed

The following committee members were present at the closed sessions of the event

Perry, Heidi
Bowcutt, Kevin
Graef, Katherine J.
Grant, Conrad
Greer, Ed
Lewis, Terry P.
Sciarretta, Al

The following topics were discussed in the closed sessions:

Range test and evaluation and threat replication abilities.

Date of posting of Closed Session Summary:

January 19, 2023

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Meeting Closed

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #4

January 27 - 28, 2022

Past

About

In lieu of our planned classified workshop being rescheduled, staff and committee members will be convening in Washington, DC to receive unclassified briefings and hold writing sessions.

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Heidi Perry
Kevin Bowcutt
Katherine Graef
Conrad Grant
Ed Greer
Michael Hamel
Dana "Keoki" Jackson
Terry Lewis
Al Sciarretta
Mitchell Simmons

The following topics were discussed in the closed sessions:

Topics discussed are space test and evaluation, threat replication, and the space range as an exemplar of the challenges and needs of future OT&E

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Location

Keck Center

500 5th St NW

Washington DC 20001, USA

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure Meeting (TRMC)

December 15, 2021

10:00 AM - 1:00 PM (EST)

Past

About

Committee Members will meet the the leadership of the Test Resources Managment Agency to discuss the study effort and DoD range needs.

Additional Information

Is it a Closed Session event?

Partially Closed

The following committee members were present at the closed sessions of the event

Conrad Grant, Ed Greer, Mike Hamel, Heidi Perry, Al Sciarretta, Mitchell Simmons

The following topics were discussed in the closed sessions:

TRMC Discussion
Study Report Outline

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure Site Visit - Edwards Air Force Base

December 15, 2021

9:00 AM - 4:00 PM (EST)

Past

About

A small subset of the committee will travel to Edwards Air Force Base in California to meet with members of the Air Force operational test community. Edwards AFB is one of the Air Force's primary air combat test centers. By visiting the site and engaging directly with Air Force operational testers, the committee hopes to gain valuable insight into their work and the challenges facing the range the Air Force test community writ large.

Additional Information

Is it a Closed Session event?

Not Closed

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure Site Visit - Point Mugu

December 14, 2021

9:00 AM - 4:00 PM (EST)

Past

About

A small subset of the committee will travel to Point Mugu Naval Station in California to meet with members of the Navy's operational test and evaluation community. Point Mugu serves as both an air and sea test range for the Navy. This will afford the committee members insight into the Navy's challenges in testing in those domains as well as their capacity to test systems across the air and sea domain.

Additional Information

Is it a Closed Session event?

Not Closed

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure Site Visit - Ft. Meade

December 13, 2021

10:00 AM - 2:00 PM (EST)

Past

About

A small subset of the committee travelled to Ft. Meade, Maryland to visit the Joint Interoperability Test Center (JTIC). The goal of the site visit was to meet with members of the testing community to develop a first hand understanding of their facilities. As the leading organziation tasked with ensuring interoperability between the servcies, JTIC can provide invaluable insight to key challenges of the joint force.

Additional Information

Is it a Closed Session event?

Not Closed

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Closed

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #3

October 27 - 29, 2021

Past

About

A data-gathering session for a study assessing the operational suitability of DoD Test and Evaluation ranges and infrastructure. The committee will hear input from invited speakers, review materials, develop a report outline, and discuss the progress of the study.

Additional Information

Is it a Closed Session event?

Institution Closed

The following committee members were present at the closed sessions of the event

Heidi Perry, Katherine Graef, Conrad Grant, Ed Greer, Michael Hamel, Dana Jackson, Terry Lewis, Al Sciaretta, Mitchell Simmons

The following topics were discussed in the closed sessions:

TETRA discussion

Date of posting of Closed Session Summary:

January 12, 2023

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Meeting Closed

Inter-Service Discussions about Advanced Battle Management System and Project Overmatch: A Meeting of Experts

October 20 - 21, 2021

Past

About

A two day closed meeting to be held at the Air Force's Common Mission Control Center (CMCC).

Organizing Project

Inter-Service Discussions about Advanced Battle Management System and Project Overmatch: A Meeting of Experts

Contact

Ellen Chou

(202) 334-2439
echou@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Webcast

Public Briefing for the Report: Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems

October 13, 2021

11:00 AM - 11:00 AM (EDT)

Past

Materials

About

Department of Defense test ranges are essential for assessing the effectiveness of military systems in realistic operating conditions across land, sea, air, space, and cyberspace. However, aging range infrastructure and limited capacity present challenges for testing advanced technologies and deploying reliable weapon systems to the field in a timely manner.

A new National Academies' report, Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight, evaluates the physical and technical suitability of DoD test ranges and infrastructure. The report, commissioned by the DoD Director for Operational Test and Evaluation, includes a comprehensive set of findings and recommendations to address range deficiencies for testing technologies anticipated to arrive between now and 2035.

We invite you to download the report and join us for a public briefing webinar on Wednesday, October 13, 2021 from 11am-12pm ET to learn more. During the webinar, members of the committee will discuss the report's key recommendations and take questions from the audience.

Additional Information

Is it a Closed Session event?

Not Closed

Organizing Project

Assessing the Physical and Technical Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Contact us

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #2

October 7 - 8, 2021

Past

Agenda

Download the Agenda

About

The second meeting of the committee on Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure. This will primarily be a planning meeting for a workshop to be held later in October and the rest of the study.

Additional Information

Is it a Closed Session event?

Partially Closed

The following committee members were present at the closed sessions of the event

Brian Holmes, Heidi Perry, Kevin Bowcutt, Conrad Grant, Ed Greer, Michael Hamel, Terry Lewis, Al Sciaretta, Mitchell Simmons

The following topics were discussed in the closed sessions:

Planning for workshop; study status update

Date of posting of Closed Session Summary:

January 12, 2023

Organizing Project

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure

Contact

Steven Darbes

sdarbes@nas.edu

Disclaimer

It is essential to the National Academies mission of providing evidence-based advice that participants in any of our meetings or events avoid political or partisan statements or commentary and maintain a culture of mutual respect. The statements and presentations during our meetings or events are solely those of the individual participants and do not necessarily represent the views of other participants or the National Academy of Sciences, which is a non-partisan, tax exempt organization that includes under its Charter the National Academy of Engineering and National Academy of Medicine, and that operates the National Research Council.

Log In

Sign in to access your saved publications, downloads, and email preferences.

Former MyNAP users: You'll need to reset your password on your first login to MyAcademies. Click "Forgot password" below to receive a reset link via email. Having trouble? Visit our FAQ page to contact support.

Members of the National Academy of Sciences, National Academy of Engineering, or National Academy of Medicine should log in through their respective Academy portals.

Email (required)
Password (required)

Don't have an account?

Register

Email (required)
New password (required)
Retype password (required) Password must be at least 8 characters
I accept the terms of use
terms of use

Thank You

Thank you for creating a MyAcademies account!

Enjoy free access to thousands of National Academies' publications, a 10% discount off every purchase, and build your personal library.

Forgot Password

Enter the email address for your MyAcademies (formerly MyNAP) account to receive password reset instructions.

Reset Requested

We sent password reset instructions to your email . Follow the link in that email to create a new password. Didn't receive it? Check your spam folder or contact us for assistance.

Reset Password

New password (required)
Retype password (required) Password must be at least 8 characters

Reset Password

Your password has been reset.

Log In

Verify email

Verify Your Email Address

We sent a verification link to your email. Please check your inbox (and spam folder) and follow the link to verify your email address. If you did not receive the email, you can request a new verification link below

Subscribe

First Name
Last Name
Email (required)
I agree to receive emails from the National Academies (required)

We respect your privacy and will never share your information.

My Academies

Naval Studies Board

Testing, Evaluating, and Assessing Artificial Intelligence-Enabled Systems under Operational Conditions for the Department of the Air Force Kickoff Meeting #1, Part 1

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #7

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #6

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Workshop

Assessment of Partnership Options for a Small Satellite System for Collecting Scientific Quality Oceanic and Coastal Data - Report Dissemination

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #5

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #4

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure Meeting (TRMC)

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure Site Visit - Edwards Air Force Base

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure Site Visit - Point Mugu

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure Site Visit - Ft. Meade

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #3

Inter-Service Discussions about Advanced Battle Management System and Project Overmatch: A Meeting of Experts

Public Briefing for the Report: Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems

Assessing the Operational Suitability of DoD Test and Evaluation Ranges and Infrastructure - Meeting #2