2
Introduction

This introductory chapter describes the assessment process conducted by the National Academies of Sciences, Engineering, and Medicine’s Army Research Laboratory Technical Assessment Board (ARLTAB). It then describes the preparation and organization of the report, the assessment criteria, and the approach taken during the report preparation.

The ARLTAB is guided by the following statement of task:

The charge of the ARLTAB is to provide assessments of the scientific and technical quality of the ARL. These assessments include the development of findings and recommendations related to the quality of ARL’s research, development, and analysis initiatives. For the 2023 assessment, the ARLTAB was charged to review the work in ARL’s four competencies: electromagnetic spectrum sciences, energy sciences, mechanical sciences, and military information sciences.

To conduct its assessments, the ARLTAB was assisted by four National Academies’ panels, each of which focused on one of ARL’s competencies. The ARLTAB’s assessments are commissioned by ARL. For this assessment, the ARLTAB consisted of 10 leading scientists and engineers whose collective experience spans the main topics within ARL’s 2023 scope. Four panels, each of which focuses on one or more of ARL’s research competencies, report to the ARLTAB. Four of the ARLTAB members serve as chairs of these panels. The panels range in size from 8 to 10 members, whose expertise is carefully matched to the technical fields covered by the areas that they review. In total, 41 members participated in the reviews that led to this report. All panel and ARLTAB members participate without compensation.

The National Academies appointed the ARLTAB and panel members with an eye to assembling a slate of experts without conflicts of interest and with balanced perspectives. The experts include current and former executives and research staff from industrial research and development (R&D) laboratories, leading academic researchers, and staff from national laboratories and federally funded R&D centers. Seventeen are members of the National Academy of Engineering, and four are members of the National Academy of Sciences. A number have been leaders in relevant professional societies. The ARLTAB and its panels are supported by National Academies staff, who interact with ARL on a continuing basis to ensure that the ARLTAB and the panels receive the information they need to carry out their assessments. The ARLTAB and panel members typically serve for finite terms, generally 1.5 to 3 years, so that

viewpoints are regularly refreshed and the expertise of the ARLTAB and panel members continues to match ARL’s activities. Biographical information on ARLTAB members appears in Appendix B.

PREPARATION OF THIS REPORT

The National Academies have been producing assessments of ARL since 1996. This assessment report answers the nine assessment criteria questions listed in Appendix C and below as they are applied to each of the four competencies (electromagnetic spectrum sciences, energy sciences, mechanical sciences, and military information sciences). Chapters 3 through 6 focus on the electromagnetic spectrum sciences, energy sciences, mechanical sciences, and military information sciences competencies areas, and Chapter 1 provides conclusions, recommendations, actionable findings and opportunities for these four competencies, along with conclusions and recommendations that address cross-cutting themes identified within these four competencies. The rest of this chapter explains the rich set of interactions that supports those judgments.

This review is based on a large amount of information received from ARL and on interactions between ARL staff and the ARLTAB and its panels. The amount of information that is funneled to the ARLTAB, including the evaluations by the recognized experts who make up the ARLTAB’s panels, provides a solid foundation for a thorough peer review. Most of the information exchange occurs during the annual meetings convened by the respective panels at the appropriate ARL sites, both in formal interactions during technical presentations and in less formal interactions during poster sessions and during joint lunch or dinner sessions.

For the 2023 assessment, ARL displayed a very healthy level of information exchange and acceptance of external comments. The assessment panels and the ARLTAB engaged in many constructive interactions with ARL staff during their virtual meeting and site visits in 2023. Each panel’s review meeting lasted 2.5 days, during which time the panel members received a combination of overview briefings by ARL management and technical briefings by ARL staff. Prior to the meetings, the panels received extensive materials for review. Useful collegial exchanges took place between panel members and individual ARL investigators during the site visits as ARL staff members sought clarification about panel comments or questions and drew on panel members’ contacts and sources of information.

The overview briefings brought the panels up to date on the broad scope of ARL’s scientific and technical work and its reorganization in October 2022. This context-building step was needed because the panels are purposely composed of people who, while experts in the technical fields covered by ARL’s competencies that they reviewed, were not engaged in collaborative work with ARL. Technical briefings for the panels focused on R&D goals, strategies, methodologies, and results of selected projects at the laboratory. Briefings were targeted at coverage of a representative sample of each of ARL’s research core competencies within their respective competencies. Briefings included poster sessions that allowed direct interaction among the panelists and staff of projects that were not covered in the briefings.¹ The panels also received Controlled Unclassified Information (CUI).

Ample time during both the overview and the technical briefings was devoted to discussion, which enabled panel members to pose questions and ARL staff to provide additional technical and contextual information to clarify panel members’ understanding. The panels also devoted sufficient time to closed-session deliberations, during which they developed findings and identified important questions or gaps in panel understanding. Those questions or gaps were discussed during follow-up sessions with ARL staff so that each panel was confident of the accuracy and completeness of its assessments. Panel members continued to refine their findings, conclusions, and recommendations during written exchanges among themselves after the meetings.

___________________

¹ Agendas of the public portions of the panel meetings can be found at the National Academies’ website at https://www.nationalacademies.org/lab/laboratory-assessments-board, accessed February 4, 2023.

In addition to the insights that they gained from the panel meetings, ARLTAB members received exposure to ARL and its staff at the ARLTAB June virtual meeting. The 2023 ARLTAB meetings refined elements of the assessment process focused on ARL’s competencies, including a discussion with ARL on its needs for the 2023 assessments, review agendas, and expertise required within the panels.

ASSESSMENT CRITERIA

During the assessment, the ARLTAB and its panels considered the following requests posed by the ARL director. These requests form the basis of the entire assessment:

1. Please comment on the scientific quality of the research within the competency, particularly in comparison to that of other leading research institutions and funding agencies, both nationally and internationally.
2. Please comment on how the competency demonstrates and reflects a broad understanding of the underlying science and research conducted elsewhere.
3. Please comment on the qualifications of the teams of ARL and extramural scientists and engineers supporting the competency.
4. Please comment on the quality, appropriateness, and utilization of facilities, equipment, information technology, and digital infrastructure.
5. Please comment on the soundness of the competency’s research methods and methodologies (e.g., the reasoning; the use of theory, modeling and simulation, analysis, and experimentation).
6. Please comment on opportunities, if found, within the competency where improved direction, increased focus, connections with other research lines, or other changes could better and more quickly address the competency objectives.
7. Please comment on any specific areas, if found, where the research may be at major risk of not meeting its objectives and provide reasoning.
8. Please comment on the overall balance of the competency’s research portfolio (e.g., core competencies, partnerships, supporting extramural partners, etc.) to address the cumulative competency goals.
9. Please comment on any promising new areas of research or novel research approaches the competency should pursue.

* Specifically excluded from the assessment criteria are the relevance to, and impact of, the scientific and technical work with respect to Army mission.

The answers to these requests were reorganized under the following section headers for Chapters 2 through 4:

• Technical Quality of the Core-Competency Research (answers requests a, b, e, f, g, h, and i)
• Competency Portfolio of Scientific Expertise (answers request c)
• Competency Facilities and Resources (answers request d)
• Overall Scientific Quality of the Competency (an overarching summary at the competency level containing elements of the assessment criteria)
• Conclusions and Recommendations (contains the most salient and actionable findings from the chapter)

APPROACH TAKEN DURING REPORT PREPARATION

This report represents the ARLTAB’s consensus findings and recommendations, developed through deliberations, which included consideration of the notes prepared by the panel members summarizing their assessments. As mentioned, the chair of each panel is a member of the ARLTAB, and in that capacity they bring their detailed knowledge of their panel’s site visit and deeper discussions. The ARLTAB’s aim with this report is to provide guidance based on the assessment criteria to the ARL director that will help ARL sustain its process of continuous improvement, as well as guidance to those managing the competencies. To that end, the ARLTAB examined its extensive and detailed notes from the many ARLTAB panel and individual interactions with ARL during 2023. From those notes, it distilled a shorter list of the main trends, opportunities, and challenges that merit attention at the level of the ARL director and the management team. The ARLTAB used that list as the basis for this report. Specific ARL projects are used to illustrate these points in the following chapters when it is helpful to do so, but the ARLTAB did not aim to present the director with a detailed account of interactions with bench scientists. The draft of this report was subsequently honed and reviewed according to the National Academies’ procedures before being released.

The ARLTAB applied a largely qualitative rather than quantitative approach to the assessment. The approach of the ARLTAB and its panels relied on the experience, technical knowledge, and expertise of its members, whose backgrounds were carefully matched to the four competency areas that were the focus of the 2023 review. The ARLTAB and its panels reviewed selected examples of the scientific and technological research performed by ARL; it was not possible to review all ARL programs and projects exhaustively. Given the necessarily non-exhaustive nature of the review process, the omission of mention of any particular program or project should not be interpreted as a negative reflection on the omitted program or project.

In line with the assessment criteria, ARLTAB’s goal was to identify and report salient examples of accomplishments and opportunities for further improvement with respect to the technical merit of ARL work, opportunities to expand its portfolio, and specific elements of ARL’s resource infrastructure that are intended to support the technical work. Both the Army Research Office’s (ARO’s) and the Army Research Directorate’s (ARD’s) extramural and intramural work exist, and often integrate, together under the holistic umbrella of each competency and its core competencies or research thrusts. Under the advisement of ARL, the approach to this assessment was not to tease out what was the ARD’s versus the ARO’s research, but rather to review the competency and its core competencies without specific focus on whether the research originated from ARO or ARD. In all cases when the work is described as “extramural” in the upcoming chapters, it is important to note that such work is still considered by ARL to be “intramural” because ARL is overseeing this work with its extramural partners. Collectively, these highlighted examples for each ARL research competency and its core competencies are intended to portray an overall impression of the laboratory while preserving useful mention of suggestions specific to projects and programs that the ARLTAB considered to be of special note within the set of those examined.

REPORT CONTENT

This chapter has addressed assessment process used by the ARLTAB and its four panels. Chapter 1 provided all conclusions, recommendations, and actionable findings found across the report. Chapters 3 through 6 provide detailed assessments of the competencies reviewed during 2023. The appendixes provide ARL’s research core competencies, biographical information on the ARLTAB members, the assessment criteria used by ARLTAB and its panels, the statement of task, a multi-modality data fusion visualization for the systems and signal processing research thrust within the electromagnetic spectrum sciences competency, and a list of acronyms found in this report.