AI chatbots that answer medical queries could save doctors鈥� time, but also run the risk of making recommendations that harm the people seeking advice.
at Harvard University and his colleagues conducted an experiment in which six oncologists addressed a variety of questions from 100 fictional people with cancer, who the doctors knew were not real. The questions were presented via a hospital electronic messaging system.
Each oncologist was asked to provide written responses to questions related to 26 scenarios, for example, someone with lung asking if they should be concerned that they are more fatigued than normal. They were then given 26 written responses to the same scenarios that were produced by GPT-4, the large language model (LLM) powering ChatGPT.
Advertisement
The oncologists were asked to assess the usefulness of the AI chatbot鈥檚 responses, not knowing they were generated by artificial intelligence. In 77 per cent of cases, the doctors felt these responses would improve their own efficiency in addressing the queries.
However, 7.1 per cent of the AI responses were judged by the doctors to have the potential to cause severe harm, such as not suggesting someone goes to hospital if they show signs of a life-threatening complication. For 0.6 per cent of the responses, the doctors believed a person could die if they followed the AI鈥檚 response. OpenAI, the firm behind GPT-4, declined to comment on the research.
When asked who they thought wrote the AI responses, the doctors suspected that only 3 out of every 10 were produced by a person, based on how they were written.
鈥淚t鈥檚 helpful to have something to help draft your responses, but we need to understand whether it actually impacts clinical decision-making and impacts care,鈥� says Chen.
While the doctors thought the AI responses would be helpful most of the time, the number of harmful or even life-threatening responses is worrying, says team member , who is also at Harvard University. 鈥淲e still need a human in the loop because there were these cases where, if unedited, it could cause harm.鈥�
In a second part of the study, the oncologists were asked to redraft the AI-written responses. At 169 words on average, these responses were far longer than the physician-written ones, at 34 words. Redrafting reduced the word count slightly to an average of 160 words. One criticism of LLM-powered chatbots is that they often produce overly verbose answers with irrelevant information.
, a family doctor in Oxford, UK, wonders whether AI-based responses would be welcomed by people seeking answers about their condition. 鈥淚 suspect most would prefer a briefer response from a clinician who knows them to a more complete, AI-generated response, even assuming the latter was safe.鈥�
Although OpenAI鈥檚 usage policy says its models mustn鈥檛 be used to provide 鈥渋nstructions on how to cure or treat a health condition鈥�, Salisbury is also worried that using AI, with its flaws, could become a crutch for doctors. 鈥淚f, after further development, most of the time AI produces an acceptably safe response, is there a risk that busy doctors will begin to rely on it and dangerous advice will slip through?鈥�, she says.
arXiv
