A French health insurance company has tested its medical chatbot with hundreds of people, in the largest real-world trial of a medical AI of its kind.
Technology companies have long promised that their AIs can help ease pressure on doctors by providing accurate medical advice, but critics have been sceptical about their accuracy and potential risks, and they have failed to find wide success. One of the most prominent AI healthcare companies, Babylon Health, went bankrupt last year.
Recent chatbots powered by large language models, such as ChatGPT, appear to offer better accuracy and fluency when providing medical advice than older AI models, and they have shown promise in closed-doors testing. But there is still scant data for how they might perform with real patients, in part because of the large risks and ethical problems of potentially incorrect or harmful advice.
Advertisement
Now, at French health insurance company Alan and his colleagues have tested their AI chatbot, called Mo, in hundreds of real-world conversations. While using Alan’s existing online medical advice service, where people can talk with real doctors over text, people were given the option to talk with Mo instead. “We’re the first one to ship the real thing in real conditions and to see how it behaves in front of patients,” says Lizée.
Rather than build their own medical chatbot, which some companies like Google have done, Lizée and his team tested several commercial models from companies like OpenAI and Anthropic to power Mo. They evaluated the models on a test made from hundreds of French medical exams to assess which models might fare better in specific scenarios. They also used anonymised conversations that Alan customers had had with doctors and tasked Mo with the same queries, comparing its responses with the doctors’.
To answer patients’ queries, Mo draws on the best model for each task based on the strengths and weaknesses identified in these tests. The team gave customers the option to have their queries answered by the AI, and those who accepted were randomly assigned to either Mo or a real doctor. To minimise potential risks, Mo had restrictions on topics it couldn’t talk about, including mental health and emergency requests. It also had each of its messages reviewed by a real doctor within 15 minutes of being sent.
The results, which included 926 conversations in total, showed that people who had spoken with the AI reported slightly higher satisfaction and “clarity of information” compared with those who spoke with a doctor. Out of 1265 messages sent by Mo, 95 per cent were rated positively by doctors, 3.6 per cent were rated as poor, and one conversation was flagged for “potentially dangerous inaccuracies” and subsequently hidden from the recipient. The content of the flagged message can’t be disclosed due to privacy reasons, says Lizée.
“You can say it’s just an individual case, but it could be extremely detrimental to that one person,” says at the University of Oxford. “From a safety point of view, this doesn’t seem ready at all, and it would be good to know what the dangerous information was. That transparency factor is really important also, to build trust and to know how people are working on improving these systems.”
While the metrics of patient satisfaction being tested by Lizée and his team are useful, they are only relevant for this particular chatbot, says at the University of Surrey, UK. “It’s difficult to generalise to medical chatbots generally,” he says. “You have a very wide range of what [a chatbot] might be dealing with, from someone scheduling an appointment to someone asking a clinical question or for advice on taking medication.”
Lizée and his team say that they plan to continue testing Mo with real people in a supervised manner, aiming to increase the number of responses rated as positive to 99.5 per cent and decrease the number of potentially harmful results to 1 in 10,000. “We needed to see the right kind of data to go to the next phase, especially in terms of safety,” says Lizée. “We’re going to continue applying these things. We’re not going to send it to all French people tomorrow unsupervised.”
But more focused and monitored testing with a group of people in a strict research environment should be carried out before rolling out the AI to the wider public, says Green. “At the moment, I’m not convinced from what I saw in this paper that it’s a good idea to roll it out at all. I think it would be unsafe to do. There should be more of a development phase here.”
arXiv
