AI surpassed doctors by 4.6 times in the new medical test HealthBench
OpenAI presented the HealthBench language model evaluation system, which sets new standards for measuring the effectiveness of artificial intelligence systems in the medical field.
The tool was developed in collaboration with 262 practicing physicians from 60 countries worldwide. Such broad geographical coverage allows for consideration of various approaches to diagnosis and treatment characteristic of different medical schools and cultural contexts.
HealthBench is based on an extensive database of 5000 clinical scenarios modeled on real medical cases. The methodology’s distinctive feature lies in its comprehensive approach. Instead of isolated questions, synthetic dialogues between an assistant and a user are used, simulating real communication in a clinical environment.
The benchmark’s multilingualism provides a truly global assessment of artificial intelligence. This is critically important for medical systems that must function in different linguistic environments without losing accuracy.
Models are evaluated on 5 key parameters. Accuracy of provided information, completeness of response, understanding of context, quality of communication, and adherence to instructions. Such multifactorial analysis allows identification of strengths and weaknesses of each artificial intelligence system.
The test results demonstrate a significant gap between the capabilities of artificial intelligence and humans. The most effective model o3 achieved a score of 60%, followed by Grok 3 with 54% and Gemini with 52%. For comparison, practicing physicians without artificial intelligence support demonstrate a result of about 13%.
Medical specialists also experience difficulties even when attempting to improve artificial intelligence responses. While with previous generation models doctors could slightly improve the quality of answers, with the newest systems the situation has changed. Human editing of latest generation artificial intelligence responses actually reduces their quality.
I think the quantitative gap between artificial intelligence and doctors’ indicators is too large to be explained by methodological features of testing. 60% versus 13%. Considering that the benchmark was developed with the participation of the medical professionals themselves.
