AI Chatbots Are Even Worse at Giving Medical Advice Than We Thought

It’s tempting to think that an LLM chatbot can answer any question you pose it, including those about your health. After all, chatbots have been trained on plenty of medical information, and can regurgitate it if given the right prompts. But that doesn’t mean they will give you accurate medical advice, and a new study shows how easily AI’s supposed expertise breaks down. In short, they are even worse at it than I thought.

In the study, researchers first quizzed several chatbots about medical information. In these carefully conducted tests, ChatGPT-4o, Llama 3, and Command R+ correctly diagnosed medical scenarios an impressive 94% of the time—though they were able to recommend the right treatment a much less impressive 56% of the time.

But that wasn’t a real-world test for the chatbots medical utility. 

The researchers then gave medical scenarios to 1,298 people, and asked them to use an LLM to figure out what might be going on in that scenario, plus what they should do about it (for example, whether they should call an ambulance, follow up with their doctor when convenient, or take care of the issue on their own).

The participants were recruited through an online platform that reported it verifies that research subjects are real humans and not bots themselves. Some participants were in a control group that was told to research the scenario on their own, and not using any AI tools. In the end, the no-AI control group did far better than the LLM-using group in correctly identifying medical conditions, including most serious “red flag” scenarios. 

How a chatbot with “correct” information can lead people astray

As the researchers write, “Strong performance from the LLMs operating alone is not sufficient for strong performance with users.” Plenty of previous research has shown that chatbot output is sensitive to the exact phrasing people use when asking questions, and that chatbots seem to prioritize pleasing a user over giving correct information. 

Even if an LLM bot can correctly answer an objectively phrased question, that doesn’t mean it will give you good advice when you need it. That’s why it doesn’t really matter that ChatGPT can “pass” a modified medical licensing exam—success at answering formulaic multiple choice questions is not the same thing as telling you when you need to go to the hospital.  

The researchers analyzed chat logs to figure out where things broke down. Here are some of the issues they identified:

• The users didn’t always give the bot all of the relevant information. As non-experts, the users certainly didn’t know what was most important to include. If you’ve been to a doctor about anything potentially serious, you know they’ll pepper you with questions to be sure you aren’t leaving out something important. The bots don’t necessarily do that.

•  The bots “generated several types of misleading and incorrect information.” Sometimes they ignored important details to narrow in on something else; sometimes they recommended calling an emergency number but gave the wrong one (such as an Australian emergency number for U.K. users).

• Responses could be drastically different for similar prompts. In one example, two users gave nearly identical messages about a subarachnoid hemorrhage. One response told the user to seek emergency care; the other said to lie down in a dark room. 

• People varied in how they conversed with the chatbot. For example, some asked specific questions to constrain the bot’s answers, but some let the bot take the lead. Either method could introduce unreliability into the LLM’s output.

• Correct answers were often grouped with incorrect answers. On average, each LLM gave 2.21 answers for the user to choose from. People understandably did not always choose correctly from those options. 

Overall, people who didn’t use LLMs were 1.76 times more likely to get the right diagnosis. (Both groups were similarly likely to figure out the right course of action, but that’s not saying much—on average, they only got it right about 43% of the time.) The researchers described the control group as doing “significantly better” at the task. And this may represent a best-case scenario: the researchers point out that they provided clear examples of common conditions, and LLMs would likely do worse with rare conditions or more complicated medical scenarios. They conclude: “Despite strong performance from the LLMs alone, both on existing benchmarks and on our scenarios, medical expertise was insufficient for effective patient care.”