Artificial intelligence is predicting the outcome of court cases about human rights violations.
This adds to the growing role of AI in law. From next year, robot judges are set to be introduced in to moderate disputes of less than €7,000. In the US, algorithms are being used in , and lawyers also use AI to analyse texts to answer legal questions and identify relevant past court case decisions.
Nikolaos Aletras at the University of Sheffield in the UK and his colleagues used four AIs to analyse 11,500 cases from the European Court of Human Rights.
Advertisement
Based on the information of a case, including allegations, the applicant and defendant’s arguments, and relevant domestic law,  the AIs were tasked with predicting whether the court would rule that a human rights violation had occurred. The best-performing algorithm correctly forecast the court’s ruling with an F1 score of 82 out of 100, a measure of how accurate its predictions were.
“Lawyers could use these models to estimate the likelihood of winning a case,” says Aletras. It could be used to help them adjust a case strategy, or decide whether cases should be prosecuted in court.
If one of these algorithms were to actually be used to make court rulings, it is possible that cases could be written in a particular way to achieve a certain outcome, says Aletras. “We need to make sure our systems can be trusted and can be explainable,” he says.
The cases involved rulings about whether defendants had breached protocols or articles of the European Convention of Human Rights, which include the prohibition of torture and the right to freedom of expression.
The AIs were also tested on their ability to predict which specific human rights articles had been violated, as well as how important the case was likely to be. The European Court of Human Rights assigns ratings on 4-point scale to the cases it sees – 4 being an unimportant case and 1 being a key case that establishes legal precedent.
The best-performing AI was only, on average, 0.527 off in its predicted importance rating. But the AIs were far less successful in predicting which, if any, human rights articles had been violated, with the F1 score dropping to 60. This could be because some human rights violations appeared on the cases it was tested on, but not in the cases it was trained on.
The team also tested whether bias arose from the AIs associating certain case characteristics – for example, a particular country – with human rights violations. They tested the algorithms’ accuracy after they had removed proper nouns such as location names – and the best-performing AI’s F1 score only dropped from 82 to 80.1.
Prejudiced AI in the legal setting can have dire consequences. One system, used by US courts in making sentencing decisions, has been heavily criticised for predicting that black defendants pose a higher risk of reoffending than they actually do.
A barrier to applying such AIs in practice is that they don’t generate an explanation for how they reach a predicted outcome, says Kevin Ashley at the University of Pittsburgh in the US. “These methods are not analysing the cases the way a lawyer would.”
Reference: arXiv,
