A US courtroom experiment suggests a popular risk assessment algorithm makes harsher recommendations than human judges – possibly because it is worse than people at anticipating which defendants will violate pretrial agreements.
“Some jurisdictions wanted to work with us to evaluate whether these recommendations are actually helping judges make a better decision,” says at Harvard University.
In the US criminal justice system, judges determine whether defendants will await trial at home or in jail. Tools like the algorithm (PSA) can help them decide by providing risk scores for whether a released defendant will fail to appear at future court dates or even be arrested for new criminal activity before their trial date. The PSA is currently used in counties and states representing a combined population of more than 56 million people, including in Arizona, New Jersey, Houston in Texas and San Francisco.
Advertisement
Imai and his colleagues assessed the PSA in a courtroom experiment in Dane County, Wisconsin. They evaluated the accuracy of human judges – either working alone or with access to PSA recommendations – at guessing defendants’ risk of failing to return for scheduled court dates or posing a threat to public safety.
These risks were reflected in how judges set conditions for allowing defendants to avoid pretrial detention: low-risk cases might merely sign an agreement to return, whereas higher-risk ones might need to make a payment called cash bail, which they would get back when they returned to court.
The researchers also used statistical methods to indirectly compare the algorithm’s predictions with human judgment. They found the PSA tool’s accuracy was worse than that of humans – and the tool’s errors were especially notable in cases featuring non-white defendants. For people of colour, the algorithm recommended unnecessary cash bail by 10 to 20 percentage points more, on average, than human judges did. Such harsh recommendations impose financial hardship on defendants who can afford to make bail and force those who can’t pay to remain in jail unnecessarily.
Judges aligned with the PSA recommendations almost 70 per cent of the time, regardless of whether they consulted the algorithm. But in cases of disagreement, the PSA tool typically made harsher recommendations, usually opting to impose cash bail.
This makes sense because the tool is “intentionally and uniquely limited”, says at Northwestern University in Illinois: it only calculates risk factors based on age and criminal history records, whereas other risk assessment tools also consider information from interviews conducted by pretrial officers.
“While a human judge might be inclined to ignore a year-long incarceration stint for a drug charge from the 1990s because it’s irrelevant to the case at hand, the PSA will most certainly account for that charge to calculate a defendant’s risk,” says Esthappan.
This experiment represents “rigorous” research, but contradicts multiple studies that found risk assessment tools such as the PSA can improve the accuracy of human judgment, says at Policy Research Associates, a behavioural health research firm in New York. So this latest study’s findings would need to be replicated in future research, she says.
“We think that this kind of experiment should be done to evaluate how these recommendations influence decisions and whether that’s actually helping us,” says Imai. “Knowing that will help you develop a better algorithm as well.”
arXiv
