Can a chatbot diagnose medical conditions as accurately as a human? Babylon Health, a UK artificial intelligence firm, says a version of its software can pass the standard exam taken by British family doctors, but some medics are pushing back against the claim.
Babylon’s AI is designed to give people a probable medical diagnosis through a “symptom checker” app.
It is not the only such app available, but it is the most visible one in the UK at the moment. The company has a contract with the National Health Service to replace traditional primary care services for those who want fast access to consultations by smartphone.
Advertisement
The results from the firm’s latest study, announced at a glitzy press event yesterday, sound impressive. Babylon says an unreleased version of its AI scored 81 per cent on questions from an exam set by the Royal College of GPs, the professional body for British family doctors. That compares with an average of 72 per cent for real doctors, says the firm.
In response, the Royal College of GPs released a statement saying that a chatbot will never be able to replace a doctor because of their wealth of experience and knowledge, not to mention their “gut feeling – when they just know something is wrong with a patient”.
Augmented doctors
In fact, Babylon doesn’t claim that AI will replace doctors, but merely augment them. At the press event, founder Ali Parsa said that while a computer may make a cancer diagnosis in future, we would still need a human medic to deliver it compassionately, and support the patient in weighing up their treatment options.
It’s not an implausible idea on the face of it. A different AI system was found last month to be better than human dermatologists at . Another algorithm has proved better than human radiologists at .
But such narrow tests are less daunting than the job that GPs do: constantly weighing up a multitude of different diagnoses, from the trivial to the terminal, while taking into account an array of social and cultural confounding factors.
We don’t yet know if the Babylon chatbots can replicate this feat. The firm’s study didn’t test the AI in the full exam for GPs – just some selected questions. (To be fair, the full exams are not publicly available.)
AI vs humans
It was also tested on 100 case studies created by doctors where it again outperformed humans. But the criteria used in the development of these questions and cases were chosen by doctors working with Babylon. Independent, peer-reviewed studies are needed before we can make any judgements on the AI’s performance.
At the press reception, Parsa said medics who are sceptical may be scared of being replaced or too old to get on board with the technology. In fact, many of the critics would accept there is potential to use AI to reduce load on the creaking NHS – but it’s a central tenet of medicine that all new healthcare interventions need proper independent testing before being widely rolled out.
Unfortunately there is no legal requirement for that when it comes to symptom checkers. They are classed as medical devices, which in Europe are much less strictly regulated than pharmaceuticals.
AI has been making such strides over the past few years that the day may well come when it can greatly help human medics. But let’s make sure we don’t get taken in by the hype so common to new tech ventures – not when lives are at stake.
