In The News
Report Recommends Improving CSA Scoring System
A congressionally mandated study of the way CSA scores are determined recommends improvements, ranging from better collection of data on things such as driver compensation and vehicle miles traveled by state, to replacing the system with a method using "item response theory," also known as latent trait theory.
The Safety Measurement System is used to identify commercial motor vehicle carriers at high risk for future crashes. It's the heart of FMCSA's Compliance, Safety, Accountability enforcement regime, known as CSA. The review of SMS was written into the Fixing America’s Surface Transportation (FAST) Act of 2015.
The resulting study by the National Academies of Sciences, Engineering, and Medicine recommended that over the next two years, FMCSA should develop a “more statistically principled approach” for the task, based on an item response theory (IRT) model. Also known as latent trait theory, it’s an approach that has been used for policy decisions in other areas such as hospital rankings, according to the panel.
SMS Data Shortcomings
FMCSA uses information collected primarily during roadside inspections to identify motor carriers that are operating unsafely and therefore theoretically at higher risk for future crashes. These carriers are subject to interventions from FMCSA, such as warning letters and investigation.
Since it rolled out in 2010, the report noted that the system has been criticized for, among other things:
- Using highly variable assessments
- Not accounting for crashes where the motor carrier is not at fault
- Including carriers that have very different tasks in the same peer groups
- Using measures that are sensitive to effects from one or more individual states
- Using measures that are not predictive of a carrier’s future crash frequency
- Using measures that are not reflective of a carrier’s efforts to improve its safety performance over time.
The National Academies panel reported, “We have found, for the most part, that the current SMS implementation is defensible as being fair and not overtly biased against various types of carriers, to the extent that data on MCMIS can be used for this purpose.
“However, we believe some features of SMS implementation can be improved upon, and some of the details of the implementation are ad hoc and not fully supported by empirical studies. Many of these details of implementation would be easily addressed if the algorithm currently used were replaced by a statistical model that is natural to this sort of discrimination problem. “
The report identified a number of data quality issues. It recommended that FMCSA should continue to collaborate with states and other agencies to improve the collection of data on vehicle miles traveled and on crashes, data which are often missing and of unsatisfactory quality. Including vehicle miles traveled data by state and month will enable SMS to account for varied environments where carriers travel – for example, in icy winter weather in the North. In addition, there is information available in police narratives not represented in the data used that could be helpful in understanding the contributing factors in a crash.
The report also says FMCSA should research ways of collecting data on carrier characteristics – including driver turnover rates, type of cargo, and method and level of compensation. For example, compensation levels are relevant, says the panel, contenting that "it is known that drivers who are better-compensated, and those not compensated based on miles traveled, have fewer crashes." This additional data collection would require greater collaboration between FMCSA and the states to standardize the effort and to protect carrier-specific information, it noted.
The committee said it was unable to recommend whether SMS percentile ranks should be made public, because it would require a formal evaluation to understand the consequences of public knowledge of the information. SMS data was pulled from public view in December 2015 after the FAST Act prohibited the public display of property carriers’ relative percentiles. At the same time, the agency removed the raw CSA data from public view to allow time to revise its SMS website, returning only that raw data to the website in March 2016.
What Will Happen With Safety Fitness Determination?
In March of this year, FMCSA announced it was scrapping a proposed rule that would have used the SMS data for issuing safety-fitness determinations for motor carriers. A large coalition of trucking industry groups wrote to Secretary of Transportation Elaine Chao to argue that the safety-fitness determination proposal should take a back seat to FMCSA fixing its CSA program.
In announcing its withdrawal of that proposal, FMCSA said that if it “determines changes to the safety-fitness determination process are still necessary and advisable in the future, a new rulemaking would be initiated that will incorporate any appropriate recommendations from the National Academies of Science and the comments received through this rulemaking.”