Google has built the best artificial intelligence yet for answering medical questions. The Med-PaLM AI can answer multiple-choice questions from medical licensing exams and common health queries on search engines with greater accuracy than any previous AI and almost as well as human doctors.
and his colleagues at Google first started working with an AI the company developed called PaLM, which was trained on a wide range of text types, including online documents, books and Wikipedia, to perform tasks that require understanding language. PaLM is among the largest of the language AIs created so far, using 540 billion parameters 鈥� the values tuned internally by the AI to form responses to questions or tasks.
The researchers then fine-tuned the AI by feeding it instructions on how to answer medical questions.
Advertisement
When challenged to answer multiple-choice questions from medical licence exams, this updated AI performed with 67.6 per cent accuracy 鈥� a 17 per cent improvement over the next best AI.
Finally, the researchers added special prompts that help the AI better follow instructions for answering medical questions, creating Med-PaLM.
The AI nearly matched human clinician performance when answering open-ended medical questions 鈥� such as 鈥渉ow can you pop an abscess鈥� 鈥� from Google search queries collected by the researchers. The AI鈥檚 answers matched the most widely accepted answer based on current medical knowledge 92.6 per cent of the time, while clinician answers matched 92.9 per cent of the time.
鈥淣ew technology should be at least as good as a human physician, if not better,鈥� says , an AI ethics researcher at Penn State Dickinson Law in Pennsylvania. 鈥淢any countries in the world, especially developing countries, need more doctors, and such tools have great potential to fill in at least some of the shortage gaps.鈥�
But Gerke also cautioned that a human tendency to trust convincing 鈥� but sometimes totally incorrect 鈥� AI answers can have serious consequences when a person鈥檚 health is at stake. Any medical AIs should be reviewed by government regulators before hitting the market, she says, and ideally they would be continually monitored even after commercial release.
Karthikesalingam says that such AIs are not yet safe and ready to be used in healthcare. One step toward making the AI safer involves asking human doctors to rate the likelihood that a wrong AI answer could lead to physical or mental harm. Just 5.8 per cent of the Med-PaLM AI answers were rated as potentially leading to harm, which is on par with 6.8 per cent of human clinician answers being rated as potentially harmful.
But Karthikesalingam says more work is needed on evaluating potential harms tied to specific medical scenarios and responses. To improve the AI, the researchers want to develop a way for it to clearly communicate any uncertainty in the responses it provides, says at Google. Such AIs should also clearly attribute and cite sources of information so that people can see where answers come from, he says.
Reference:
