In late September, the U. S. Department of Transportation’s Inspector General (DOT IG) published a report on the progress made by the Federal Motor Carrier Safety Administration (FMCSA) in overhauling the program it uses to assess motor carrier safety.
That program, Compliance, Safety, Accountability (CSA), and specifically its analytical component, the Safety Management System (SMS), while an improvement over prior FMCSA safety practices, showed sufficient weaknesses in its nine years of life.
On the plus side, the DOT IG noted that the corrective action plan submitted to Congress by FMCSA in 2018 addressed many of the National Academy of Sciences (NAS) recommendations, including the development by FMCSA of an “item response theory” (IRT) model to prioritize carrier safety interventions. (IRT brings mathematical validity to what had been arbitrary assessments of fleets under SMS, such as severity weights for violations.)
FMCSA completed a small scale IRT model by mid-January 2019 and had also held promised public briefings on the CSA overhaul project. On the negative side, the DOT IG found the FMCSA efforts so far lack details on actual implementation and transparency on how carrier safety rankings would be calculated. Also, there were few details on how the project would improve the carrier safety assessment process.
But the biggest news was in FMCSA’s official response to the DOT IG report. FMCSA will have a full-scale IRT model completed in fall of 2019 but will not decide on whether to adopt the IRT model until September 2020.
The reason for FMCSA’s delayed decision is data deficiency. IRT models are deemed more capable of handling variables in safety scoring than is SMS. But to do that, IRT demands more data – and FMCSA has found that data hard to come by. For example, the FMCSA Corrective Action Plan said it would investigate obtaining data on driver turnover rate, type of cargo, and method and level of driver compensation to determine whether those factors influence the IRT calculation of safety risk. But FMCSA found the industry as a whole unwilling to part with proprietary or competitive data.
The Corrective Action Plan also said FMCSA would target better data on fleet mileage (exposure) and improvements in the completeness and timeliness of crash reporting. But an analysis of utilizing fleet mileage data from the International Registration Plan (IRP) showed only a 2% improvement over current SMS exposure data, not worth the cost of the effort. On the crash reporting front, FMCSA has partnered with the National Highway Safety Administration (NHTSA) to bring uniformity to state reports, but results are in the future.
In many ways the FMCSA overhaul of CSA is like an automaker launching a new car. The IRT model looked good on the NAS drawing board. The prototype small-scale IRT model performed well in January – but that was just a few laps around the track in the hands of a professional driver. Going into full-scale commercial production is another thing entirely. Can FMCSA get adequate parts – the data? Will the new car handle all 899 current potential violations, even when consolidated, as FMCSA has done, to a mere 381 variables? And finally, will it sell? As FMCSA asks, can we explain this to the public? The answer will come in September 2020.
You can read more about FMCSA’s efforts to revamp the CSA program. See the PrePass whitepaper “CSA Overhaul Gets Underway,” as well as the blogs “FMCSA Takes First Steps to Overhaul CSA” and “CSA Follow-Up: Where the New FMCSA Safety Program is Heading”.