CHAPTER 6

Data Decision Tree for Big Data in Freight Transportation Planning and Operations

States and other planning agencies explore big data sources to facilitate better freight transportation planning and operations decision-making. Limited funds, differing planning horizons, proprietary information, insufficient human resources, and misinformation about available data have resulted in challenges related to the acquisition, integration, and use of freight-related big data.

In a review of the literature, the team identified multiple relevant sources of freight data currently in use by practitioners:

• Global Positioning System (GPS) [data provided by INRIX and the American Transportation Research Institute (ATRI)];
• Electronic logging devices (ELDs);
• Weigh-in-motion (WIM) devices [data provided by state departments of transportation (DOTs)];
• Inductive loop detector technology, which records a vehicle’s unique magnetic signature to classify trucks for tracking purposes between multiple detectors;
• Radar and computer vision technology for identifying vehicle shapes and sizes to classify trucks;
• Administrative records, such as delivery records and bills of lading, which contain information about shipments, delivery dates and times, locations, commodities, and other aspects of freight movement;
• Economic census and industry surveys; and
• Other integrated datasets that combine information from multiple sources and are published by organizations for transportation planning and operations purposes [e.g., FHWA’s Freight Analysis Framework (FAF); S&P Global Transearch; IMPLAN; Woods & Poole Economics, Inc.; FreightWaves Sonar].

These identified data sources have use cases in freight planning and operations, including

• Providing insights into industry trends, goods movement, and forecasts based on economic indicators and consumer demand;
• Identifying traffic congestion hot spots;
• Measuring travel time reliability;
• Inferring trip origins and destinations;
• Determining turning movements of vehicles at intersections and ramps;
• Understanding the effects of traffic conditions on driver behavior, truck idling, and emissions; and
• Correlating roadway conditions such as bad weather with truck incidents and overall system resiliency.

Despite the availability of these data sources, stakeholders are challenged to determine which data source to use to address specific use cases.

Current State of Practice

To better understand the current state of the practice and to gain insights into the uses, challenges, and opportunities of freight data, the team conducted interviews with state DOT staff and freight data stakeholders from Arizona, Florida, Nevada, New Jersey, Oregon, Texas, Washington State, and The Eastern Transportation Coalition (TETC). Following is a summary of findings on the current state of the use of freight data by state DOTs.

How Do DOTs Currently Use Freight Data?

• DOTs use freight data for various purposes, and the needs of high-level state plans vary from location-specific projects. Examples of freight data uses include origin–destination analysis, truck volume information, commodity flows, congestion studies, truck parking studies, truck weight studies, oversize/overweight permitting, network routing, corridor profile studies, long-range planning, and forecasting.
• Some DOTs have a centralized data system to share with others for various needs. For example, the Florida DOT’s (FDOT’s) DataLytics system takes roadway data and visually summarizes the information, making it more consumable for users. Similarly, the Texas DOT’s data are available for related studies and consultants working on those studies.
• Some DOTs or groups of DOTs (for example, TETC) may procure data and provide them to other users within the state (or group), such as metropolitan planning organizations (MPOs). The S&P Global Transearch database is an example of a dataset that a state DOT may purchase for MPOs.
• The Oregon DOT found success in communicating the importance of the freight industry to legislators through rigorous analysis, clear messaging, and readable reporting.

What Challenges Do DOTs Have When Working with Big Data?

• DOTs such as those in Florida and Oregon have in-house capabilities to process the data they acquire; however, many DOTs rely on consultants to process data to provide insights.
• A dataset may be so large that smaller MPOs are overwhelmed by it and rely on the state to postprocess and interpret the data.
• Though some DOTs purchase data directly, others do not. Consultants may purchase data through their contracts with DOTs, which results in multiple and sometimes more-expensive contract agreements for the DOTs. Consultants may have some bias for certain data providers.
• The gap between technical expertise and the traditional analyst, when the traditional analyst does not understand concepts such as weights and representative sampling, remains a challenge for DOTs.
• DOTs may also be reluctant to purchase third-party data because of funding constraints as well as challenges with validating the data, risk management, and public trust in the dataset.
• DOTs are interested in the analysis and results of the data and may lack the in-house capability to store third-party data or take responsibility for it. FDOT, for example, has a comprehensive public records law that affects data governance and storage.
• Data collection, processing, and reporting methods may change yearly, and vendors may not provide sufficient information on those changes. An example is how forecasting is performed, as this varies by industry sector.
• DOTs also deal with data compatibility issues when data from different sources with similar information do not align.

What Are Gaps and Challenges with Existing Big Data Sources?

• Some DOTs reported insufficient data to develop performance measures. For example, there are insufficient data to understand railroad movements because of privacy concerns by the Class I railroads.
• Understanding the last mile is also a challenge, although the focus of most DOTs is on state-owned highways.
• DOTs face the challenge of understanding and representing business behavior related to supply chains and shipping, especially in the aftermath of the COVID-19 pandemic.
• DOTs currently rely on federal resources for information on statewide commodity flows. Anything beyond those data is private and not publicly available, which makes it challenging to validate. Questions that remain unanswered include the following:
  • Where are commodities being moved?
  • What is their value?
  • What corridors are they moved through?
  • How does that change over time?
• Some data sources report freight demand as indices, which are not easily digestible and can misrepresent actual demand.
• With data changing often, integration into planning models, forecasting, and decision-making remains challenging.
• During data processing of WIM data, reported errors may not be actual errors; rather, they may be attributed to unclassifiable vehicles (vehicles with extra axles which WIM equipment does not recognize).

What Are Some Recommendations Regarding the Use of and Access to Big Data?

• Develop a crosswalk of the data, how it was collected and processed, what the data will tell users, and how others have used it.
• Compare and document methodologies used in collecting, processing, and reporting data sources.
• Expand the National Performance Management Research Data Set (NPMRDS) program to make the same data accessible to all DOTs for comparative analysis and cost savings and to foster data sharing across state borders.
• Understand state purchasing laws and contract vehicles and know the agency’s purchasing rules.
• Understand the agency’s information technology options for data and data maintenance costs.
• Convert information into actional insights and decision-making.

Product Development

To address the current gaps in determining the best datasets to use for a specific use case, the team developed a web-based, interactive freight data decision tree (Figure 6-1). This product, available online at https://data.transportationops.org/data-decision-tree-big-data-freight-transportation-planning-and-operations, showcases what big data sources are available to address freight planning and operations use cases. Four use cases are included in the product: origin–destination analysis, truck volumes, truck congestion, and truck parking. In addition, 11 datasets are included in the decision tree:

1. ATRI Truck GPS data,
2. FAF,
3. Freight Mobility Trends Dashboard,
4. Geotab,

5. Highway Performance Monitoring System (HPMS),
6. INRIX,
7. NPMRDS,
8. StreetLight Insight,
9. North American Transborder Database,
10. Transearch, and
11. WIM.

To build the freight data decision tree, the team used a JavaScript library that renders an embedded data structure into a collapsible tree view. A subtree expands when the user clicks on each node on the tree until the applicable list of data sources for that use case appears, as shown in Figure 6-1. When a data source is selected, the user is prompted to the section describing the content of the data. As shown in Figure 6-2 and Figure 6-3, the information presented includes the data source’s metadata and use cases identified from the literature. Table 6-1 lists the attributes and information captured for each data source.

Product Assumptions and Constraints

The freight data decision tree was developed with the following assumptions:

• The information published on the web application is limited to what was provided by the data vendors or identified during the literature review. Therefore, the information is subject to change as vendors update their respective products.
• The project panel for NCHRP Project 08-119 selected a limited number of use cases, as previously noted. Additional use cases for freight data sources exist and could be added to expand the tool.

Opportunities for Further Development

The first version of the freight data decision tree showed a proof of concept of how data practitioners can identify which data sources best serve their needs. There are opportunities to expand on this product for general adoption by DOTs. Opportunities to further expand the tool for other use cases include

• Incorporating datasets on waterway movement and other modes of transport (rail, air, pipeline, and marine);
• Developing a comparison table between multiple datasets to illustrate disparity between the datasets;
• Incorporating a limited number of data schema crosswalks to address the disparities between datasets (these crosswalks map the elements in one data schema to the equivalent element in another); and
• Showing discrepancies in results for different data vendors for the same analysis performed.

Table 6-1. Data source attributes.