Project Activities—Phase II

Phase II of the research consisted of Tasks 6 through 9 and focused on diving deeper into specific content and promotional materials through case studies; creating draft/sample digital content and guidelines; and then reviewing the content with state DOT practitioners to provide proof of concept. The final task included panel review and input on the draft content and final adoption.

In Phase II, the research team completed four deep-dive case studies, developed draft digital content for AASHTO’s Transportation Management Hub, and facilitated a proof-of-concept workshop. The research team prepared final deliverables, including digital content for inclusion on the Hub, articles for publication, and both video and in-person presentations for relevant audiences. The following supplemental materials are available on the National Academies Press website (nap.nationalacadmies.org) by searching for NCHRP Research Report 1146: Risk Management at State DOTs: Building Momentum and Sustaining the Practice:

• An Executive Summary of NCHRP Research Report 1146,
• A PowerPoint presentation to support implementation by state DOT staff,
• A risk management checklist for assessing agency maturity in risk management, and
• Appendices E through H, which contain the risk management case studies.

Deep-Dive Case Studies

Detailed case studies were developed to provide insights into organizational characteristics and practices that state DOTs can use to adopt and sustain risk management. To narrow down which agencies would be chosen, the team created a matrix (Table 1) to compare potential agencies and understand their strengths in the highlighted key elements.

From the matrix, four agencies were selected to perform deep-dive case studies: Central Federal Lands Highway Division (CFLHD); Danish Road Directorate (Vejdirektoratet); a state DOT mini-scan with UDOT and Maine DOT; and Port Authority of New York and New Jersey (PANYNJ). The research team interviewed each agency to understand their risk management processes and learn how other organizations could successfully implement them. The team compiled the interview findings to create deep-dive case studies for use by other agencies interested in what these organizations are currently doing or have previously done.

Topics discussed during each agency interview included the following:

• Overarching questions about risk management processes used within the agency at the project, program, and enterprise levels
• Data and tools used within the agency and data sharing
• Quantification of risk

• Educating staff is key.
• Cross-sectional representation (project/program) is needed to fully understand risk.
• Require risk identification/strategies and monitoring of the risks throughout the life of the project.
• Empower Project Managers to evaluate and elevate risks.

Appendix E presents the CFLHD case study.

Case Study #2: Vejdirektoratet, the Danish Road Directorate

Vejdirektoratet, the Danish Road Directorate, is responsible for developing, planning, and building road infrastructure in Denmark: motorways, a substantial portion of country and local roads, and many bridges. It helps ensure those using Danish state roads can do what they need to do and get where they need to go. Vejdirektoratet must balance mobility, environment, and road safety. Overseen by the Ministry of Transport, Vejdirektoratet’s work consists of three elements: planning, which monitors, manages, and informs about state road issues; plant and operation, which handles project management and prepares estimates of construction projects; and traffic handling and management, which deals with basic operational needs to ensure functioning and level of service for the state road network. On June 22, 2023, the research team interviewed the Chief Risk Manager for Vejdirektoratet to learn more about how it has built and maintained its risk management process.

Keys to Vejdirektoratet’s risk management efforts include the assessment of project risk using estimates informed by the professional judgment and experience of project managers, learning from past mistakes, and developing overarching risk management techniques to be used for all projects. Core elements of the risk management process include centralizing risk management in a set of dedicated professionals instead of outsourcing it, which ensures consistency of methods and metrics throughout, and having the support and understanding of top management.

Key takeaways include the following:

• Keep risk management simple; do not overcomplicate things.
• Centralize risk management.
• Embed risk management as part of project management.
• Create a risk framework and policy to show leadership buy-in.
• Learn from past mistakes to reframe and create a regime that incorporates risk management.
• Develop overarching risk management techniques for program use.

Appendix F presents the Vejdirektoratet case study.

Case Study #3: State DOT Mini-Scan (UDOT and Maine DOT)

This mini-scan consists of two state DOTs: Maine DOT and UDOT. Both house robust risk management programs, notwithstanding the size of their agencies. To learn more about how each agency has built and maintained its risk management processes, the research team interviewed the Director of Transportation Performance Management at UDOT, on June 1, 2023, and the Chief Engineer at Maine DOT, on June 13, 2023.

Keys to the two agencies’ risk management efforts include work done at the project level, addressing program risks, and managing enterprise risk with separate teams. Maine DOT manages project risk alongside the project’s life to allow for time to adjust and manage potential risks, while UDOT, in addition to identifying project risk, also investigates task-level risks. Maine DOT embeds program risks within project risks, emphasizing transparency; works to remove silos

between its programs; and fosters multidisciplinary work. Maine DOT also sees the importance of managing enterprise risk, so it created the Results and Information Office made up of leaders and directors who help identify enterprise risk for the agency. UDOT manages enterprise risk through an auditing team.

Core elements to both the Maine DOT and UDOT risk management processes include defining and integrating business practices across departments, ensuring internal communication so staff are aligned and working toward the same goal, building an executive team with top-down support, maintaining leadership buy-in and support, and having organizational structure in place.

Lessons learned include the following:

• Participate in domestic scans to learn firsthand from other state DOTs.
• Define business practices to ensure like terminology.
• Eliminate silos to create a sense of teamwork across disciplines.
• Understand and incorporate staff needs as workplace standards change.
• Develop an approach to communicate risks that are shared both internally and externally.
• Align risk where there is leadership support.

Appendix G presents the state DOT mini-scan case study.

Case Study #4: Port Authority of New York and New Jersey

The PANYNJ manages and maintains transportation and trade infrastructure critical to the New York and New Jersey urban area, including five airports, the seaport, the Port Authority Trans-Hudson rail transit system, six tunnels and bridges, bus terminals, and the World Trade Center site. On July 6, 2023, the research team interviewed the PANYNJ team to learn more about how it has built and maintained its risk management process.

Key elements to the PANYNJ risk management efforts include maintaining a dedicated agencywide enterprise risk management (ERM) system that coordinates subject matter experts (SMEs) and risk evaluation for projects using SMEs’ expertise and knowledge.

Key takeaways include the following:

• ERM is both a top-down and bottom-up process.
• Top-down risk management ensures that risks can be managed across the entire organization, while bottom-up risk management allows for the identification of new risks.
• While the ideal is to be able to quantify risk, it is also important to be nimble. If a risk is new enough, it may need to be qualitatively analyzed.

Appendix H presents the PANYNJ case study.

Digital Content Workshop

On Thursday, July 13, 2023, the research team met with the NCHRP Project 08-151 panel for an in-person digital content workshop as part of Task 7. The purpose of this workshop was to review potential digital content to be developed for the AASHTO Transportation Management Hub in Tasks 8 and 9 and to discuss immediate next steps as well as answer any questions or comments from the panel. The workshop consisted of a review and discussion of completed tasks to date, content review, examples, general format for digital content, and next steps.

Several core elements contained within the snapshots were outlined by the research team and confirmed as important during the workshop, including the following:

• Value proposition
• Quantification of risk
• Organizational change
• Data and tools
• Culture of risk
• Integration with existing processes
• Communication and promotion

The research team went through the core elements and the example content associated with each. Voting sessions were conducted to gather insight into which previously discussed proposed content the project panel believed was the most important for the end user.

The first core elements discussed were culture of risk and organizational change. Culture of risk example content included risk management policies, risk briefing memos, state DOT testimonials, and videos on culture of risk, while organizational change example content included change management activities, an organizational change management framework, a risk organization chart, and risk roles. Based on project panel voting, the most important content included the following:

• Create a link between risk management and the strategic plan.
• Create a glossary and framework.
• Present storytelling through videos and testimonials.
• Provide a pyramid of risk roles and responsibilities.
• Incorporate enterprise, program, and project areas.
• Limit words to high-level topics and big concepts to orient the user.
• Create an organization chart.
• Create an inviting landing page with information about and leading questions for risk management, where the user can decide what is worthwhile.
• Introduce examples of risk management policy.
• Show strategic plan alignment.
• Show the importance of change management training.
• Look into combining culture of risk and organizational change.
• Keep a high-level approach.

The next core elements were value proposition and communication and promotion. Value proposition example content included agency testimonials, targeted case studies, infographics and benefits, and business case guidance, while communication and promotion example content included best practices, risk communication templates, targeted case studies, training presentations, and communicating risk with partners. Based on project panel voting, the most important content included the following:

• Infographics and benefits
• Testimonials
• Best practices
• Training presentations
• Advocating for simplicity
• Targeted case studies
• Content focused on helping the user gather intelligence at any stage of development
• Communicating risk with partners and anyone else involved in risk management
• Modifiable templates and content where available

• Risk communication templates, such as the value of risk management (one-pager)
• A communication plan that speaks to value proposition, emphasizing buy-in
• Potentially combine value proposition and communication
• Business case guidance

The final core elements discussed were integration with existing processes, quantification of risk, and data and tools. Example content included standard operating procedures and risk checklists for integration with existing practices, project risk assessments, risk assessment library, alternatives analysis, program risk register, enterprise risk register for quantification of risk, risk priority map, federal data, and local data for data and tools. Based on project panel voting, the most important content included the following:

• Determine how to measure qualitatively and quantitatively.
• Direct users to already developed content.
• Combine integration with process, and include how to measure risk.
• Use inherent risk management modules in project management software and determine how that could feed into integration of existing processes.
• Show physical locations with asset risk in high-vulnerability areas.

Wireframe Review

Following the digital content workshop, the research team created a storyboard and wireframe for project panel review. After gaining concurrence on the storyboard and wireframe of the website, the research team started developing the website and conducted a focus group with state DOT practitioners to test-drive the concept.

The project panel walked through the wireframe, which laid out the concept for the landing page (Figure 3) and additional content pages (Figure 4).

The research team wanted to gather opinions on specific aspects of the wireframe, including design, tone, and content, so the project panel was asked for their feedback on a variety of questions. A summary of responses from various members is as follows:

• Research Team Question: Should the design be consistent throughout or is some variance okay?
  • – Panel Response: The organization is nice, but it would be good to distinguish sections into their own lane as you click through. There does not need to be consistency between existing and building, but within the two options, there should be some. Functionality over force is key.
• Research Team Question: Should there be an all-inclusive menu that can be accessed outside of the enterprise, program, and project areas?
  • – Panel Response: Yes, there should be a table of contents and structure of links.
• Research Team Question: What should the tone be—more casual, professional, or a mix of both?
  • – Panel Response: It is valuable to keep the tone as casual as possible, while still being professional. Simple plain speak is a good approach to begin but once you get into the details and specifics, language should line up with the resources being provided and a more professional tone would be beneficial. Finding the right mix is key.
• Research Team Question: Do you envision content being examples and resources or a lot of new content? If new content, which areas do you think would be most useful?
  • – Panel Response: The website needs to help users build risk management, where they will be provided recommendations of best practices and resources, as well as tools for users creating or sustaining risk management, examples, and other resources to continue their practice.

• Research Team Question: Do we need to provide fewer content options and make it simpler, or do we need several examples and more content?
  • – Panel Response: Have more best practices and less content to point people to starting points and to make sure the content being provided is in editable formats so users can download and modify according to their needs. Checklists should also be provided for users at the front end.
• Research Team Question: Who do you think will ultimately be the target audience?
  • – Panel Response: The website must be focused on leadership, change agents, people interested in risk management, and people doing the work.

The team evaluated feedback from the project panel during the wireframe review and incorporated it into an updated and usable platform reviewed with state DOT practitioners during the state DOT focus group.

State DOT Focus Group

On Thursday, September 28, 2023, the research team met with 11 state DOTs to test-drive developed material to verify that the digital content and promotional materials are in line with expectations and could be implemented moving forward. The purpose of this workshop was to meet with the focus group and gain feedback on the project’s digital deliverable with the goal of refining it, including content usefulness, user experience, and accessibility. To do this, the research team previewed the website and participants provided comments through an interactive platform.

To start, the research team provided a brief project overview, including the consolidation of the seven core elements into the four overarching categories believed to be what practitioners will need to build and sustain risk management programs. The categories are as follows:

• Culture of Risk and Organizational Change
• Process Improvement
• Communication and Promotion
• Data, Tools, and Risk Quantification

Participants were asked to provide comments for the landing page; enterprise, program, and project risk management pages; and the quick links page, with the guidance, examples, and tools gathered in one place.

Feedback from participants included the following:

• The visual look is good, it is clean and a great collection of resources, but consider adding more colors to distinguish content and help with better visual accessibility. The content and tools look helpful but revise the sections at the top of each role with questions to another location or better explain the purpose.
• Participants suggested a search function be included to help website visitors conduct keyword searches if they are not sure what they are looking for. Other things to consider included adding more roles, a way to determine maturity level, and a connection to elements AASHTO laid out in the subsequent pages that are foundational in the AASHTO ERM guide. Color coding by maturity level would be nice. Also, having a sidebar or bubble with varying inputs based on “superstar” agencies and how they do things could be a good addition.
• This is a great launchpad, and they want the potential for it to be refined in the future. It is valuable to see where this is going, but many wondered where this will fit within the universe of

• resources being produced. They liked the idea that it could evolve and grow with the needs of the users.
• Bringing everyone to a basic level is important, and this helps with that. It also benefits leadership by having it be role-based because they can point their staff to specific areas.
• It is a good place to start and hearing from everyone across the board will help make this better. It is a tall order to build a site that speaks to everybody, and continual progress is being made which will make a better project in the end.