Post Thumbnail

AI surpassed doctors by 4.6 times in the new medical test HealthBench

OpenAI presented the HealthBench language model evaluation system, which sets new standards for measuring the effectiveness of artificial intelligence systems in the medical field.

The tool was developed in collaboration with 262 practicing physicians from 60 countries worldwide. Such broad geographical coverage allows for consideration of various approaches to diagnosis and treatment characteristic of different medical schools and cultural contexts.

HealthBench is based on an extensive database of 5000 clinical scenarios modeled on real medical cases. The methodology’s distinctive feature lies in its comprehensive approach. Instead of isolated questions, synthetic dialogues between an assistant and a user are used, simulating real communication in a clinical environment.

The benchmark’s multilingualism provides a truly global assessment of artificial intelligence. This is critically important for medical systems that must function in different linguistic environments without losing accuracy.

Models are evaluated on 5 key parameters. Accuracy of provided information, completeness of response, understanding of context, quality of communication, and adherence to instructions. Such multifactorial analysis allows identification of strengths and weaknesses of each artificial intelligence system.

The test results demonstrate a significant gap between the capabilities of artificial intelligence and humans. The most effective model o3 achieved a score of 60%, followed by Grok 3 with 54% and Gemini with 52%. For comparison, practicing physicians without artificial intelligence support demonstrate a result of about 13%.

Medical specialists also experience difficulties even when attempting to improve artificial intelligence responses. While with previous generation models doctors could slightly improve the quality of answers, with the newest systems the situation has changed. Human editing of latest generation artificial intelligence responses actually reduces their quality.

I think the quantitative gap between artificial intelligence and doctors’ indicators is too large to be explained by methodological features of testing. 60% versus 13%. Considering that the benchmark was developed with the participation of the medical professionals themselves.

Autor: AIvengo
For 5 years I have been working with machine learning and artificial intelligence. And this field never ceases to amaze, inspire and interest me.

Latest News

The first LAARMA system protects animals on Australian roads

In Australia, animal-vehicle collisions are a serious problem for this continent's ecosystem. Now scientists have found a technological solution. The world's first roadside LAARMA system based on artificial intelligence that protects wild animals from dangerous encounters with traffic.

Nvidia introduced Cosmos model family for robotics

Nvidia company introduced the Cosmos family of AI models. Which can fundamentally change the approach to creating robots and physical AI agents.

ChatGPT calls users "star seeds" from planet Lyra

It turns out ChatGPT can draw users into the world of scientifically unfounded and mystical theories.

AI music triggers stronger emotions than human music

Have you ever wondered why one melody gives you goosebumps while another leaves you indifferent? Scientists discovered something interesting. Music created by artificial intelligence triggers more intense emotional reactions in people than compositions written by humans.

GPT-5 was hacked in 24 hours

2 independent research companies NeuralTrust and SPLX discovered critical vulnerabilities in the security system of the new model just 24 hours after GPT-5's release. For comparison, Grok-4 was hacked in 2 days, making the GPT-5 case even more alarming.