A new open benchmark is putting medical LLMs under a microscope, and the early findings are sobering. The Open Medical-LLM Leaderboard evaluates models like GPT-3, GPT-4, and Med-PaLM 2 across five healthcare datasets totaling over 200,000 questions, from USMLE-style licensing exams to PubMed abstract comprehension. The metric is blunt and unforgiving: accuracy. Either the model gets the medical answer right, or it doesn’t.
The stakes couldn’t be higher. The leaderboard’s creators point to a chilling example: when asked about treating a pregnant patient, GPT-3 recommended tetracycline while simultaneously acknowledging the drug is contraindicated during pregnancy because it can cause bone growth problems in the fetus. The model knew the danger and prescribed it anyway. In a recreational chatbot, that’s a quirk. In a clinical setting, that’s a malpractice suit waiting to happen — or worse.
The benchmark pulls from five distinct sources: MedQA (1,273 test questions from the USMLE), MedMCQA (6,100 questions from Indian medical entrance exams), PubMedQA (500 expert-labeled yes/no/maybe questions based on abstracts), and three MMLU subsets covering clinical knowledge, genetics, and anatomy. Together they test not just memorized facts but reasoning over biomedical literature and clinical decision-making.
What’s notable is how deliberately unglamorous this evaluation is. No chat interfaces, no bedside manner scoring, no simulated patient interactions. Just multiple-choice accuracy on questions that actual medical students and practitioners must answer correctly. The leaderboard’s bet is that if a model can’t reliably pass a licensing exam, it has no business anywhere near a treatment plan. Given the tetracycline example, that’s not a hypothetical concern — it’s a documented failure with a fetus on the receiving end.
