One of OpenAI’s leading artificial intelligence models can outperform humans at diagnosing medical conditions, but does that mean AI is ready to replace human doctors? Not quite, although such technology could increasingly play a role in medical treatment.
at Harvard Medical School and his colleagues at universities, hospitals and companies across the US have put o1-preview, an AI model released by OpenAI in September, through a battery of tests designed to assess its performance on a range of medical tasks. These were adapted from tests used for human doctors and included differential diagnosis generation, the process by which doctors rule out potential causes of an illness based on symptoms and testing, and reasoning in treatment and diagnosis, where doctors weigh up problems, make choices about treatment and decide what to do next.
Measured across all the tasks, the AI far outstripped both previous AI models and human performance, as measured by human experts who graded the responses. For example, in a test of clinical reasoning, o1-preview got a perfect score in 78 out of 80 cases it was set, compared with 47 out of 80 for GPT-4, an earlier OpenAI model, and 28 out of 80 for human hospital doctors. “It was inhumanely high,” says Rodman.
Advertisement
The team took care to rule out the possibility that some of the medical cases the AI was asked to judge could have been included in its training data, which would have artificially boosted its performance.
But Rodman says these high scores don’t mean we should favour AIs over human doctors, or turn to a chatbot for diagnoses – that would be “irresponsible”, he says. Instead, the researchers wanted to highlight how benchmarking tests like the ones they used aren’t up to scratch.  “I’m worried for medicine that we’re not taking these technologies seriously. We’re not developing meaningful benchmarks,” he says, and missing large parts of the diagnostic process.
For example, AI models can’t emote and understand non-verbal signs from patients that could suggest the need for further investigation in the same way human doctors can, says Keith Grimes, a former family doctor at UK-based digital health consultancy . “This paper looks at a narrow section of what a doctor does,” he says.
at the University of Oxford, a current family doctor, says that will always be a major difference between the capabilities of humans and AI. “Patients don’t necessarily present with clear-cut symptoms,” she says. “Depending on how you ask the question, different things come out.” Patients’ underlying conditions and anxiety can also change how they present that information. “Most of the diagnostic difficulties happen because there’s a great mess of symptoms, and you’ve got to try and decide what’s relevant.”
That doesn’t mean there isn’t a use for AI in medicine. “There definitely will be a point at which they outperform doctors,” says , another family doctor at the University of Oxford – just not yet. “I think we’re moving towards a complementary era, where the AI does the large data crunching stuff really well and we’re left to focus on the human side of medicine.” However, AI systems totally replacing doctors feels impractical to him – and he says people shouldn’t treat AI answers as being better than those from medical professionals.
“I don’t think there’s any harm at all in a patient going and consulting with AI, putting in their symptoms, and bringing the outputs to their doctor as the basis of conversation alongside whatever they’ve done before with Google,” says Allen. “But my doctorly advice to the readership is that it’s done with a bit of intellectual humility on both sides.”
Yet many in the medical profession – including Rodman and his colleagues – recognise the clear desire for AI doctors. “This is true in both of our systems in the UK and the US: the healthcare system is falling apart,” says Rodman. “I’m a doctor, and it takes me a long time to see my primary care physician. People are looking – understandably – at the failures of the healthcare system and really hoping that this technology will help patch them over.”
arXiv
