Offline tests showed worse results for AI models

Post Thumbnail

Artificial intelligence has officially surpassed the average human in IQ. GPT-5 Pro showed results from 110 to 138 points, while the average human IQ is 100 points. But let’s understand what this really means.

The portal Tracking AI conducted large-scale testing of AI models. At the top were 2 versions of GPT-5 Pro. With computer vision enabled and disabled. Behind them follow Gemini 2.5 Pro, Claude Opus 4 and Grok 4.

But there’s an important nuance. Unlike humans, artificial intelligence isn’t limited in time and can make up to 10 attempts to solve each task. This was done to bypass security systems that sometimes block words like exam or training.

For testing they used 2 types of tasks. First — the official Mensa Norway test, 35 tasks in 25 minutes. Second — a special offline test, created from scratch and absent from the internet. This excluded the possibility of pre-training models on these tasks.

They tested models with computer vision enabled and disabled. In the second case, tasks were described completely in text. And you know what? Almost all models showed worse results on the offline test. This proves — quality benchmarks for artificial intelligence need to be created from scratch and protected from leaks to the internet.

Artificial intelligence formally surpassed humans in classical intelligence tests. But the comparison is incorrect. IQ tests measure a specific type of pattern recognition in which artificial intelligence is naturally strong. Plus 10 attempts without time limits – these are completely different testing conditions. It looks like these “researchers” are deliberately creating conditions to make artificial intelligence look smarter.

AIvengo > Reviews > Offline tests showed worse results for AI models
Почитать из последнего
UBTech will send Walker S2 robots to serve on China's border for $37 million
Chinese company UBTech won a contract for $37 million. And will send humanoid robots Walker S2 to serve on China's border with Vietnam. South China Morning Post reports that the robots will interact with tourists and staff, perform logistics operations, inspect cargo and patrol the area. And characteristically — they can independently change their battery.
Anthropic accidentally revealed an internal document about Claude's "soul"
Anthropic accidentally revealed the "soul" of artificial intelligence to a user. And this is not a metaphor. This is a quite specific internal document.
Jensen Huang ordered Nvidia employees to use AI everywhere
Jensen Huang announced total mobilization under the banner of artificial intelligence inside Nvidia. And this is no longer a recommendation. This is a requirement.
AI chatbots generate content that exacerbates eating disorders
A joint study by Stanford University and the Center for Democracy and Technology showed a disturbing picture. Chatbots with artificial intelligence pose a serious risk to people with eating disorders. Scientists warn that neural networks hand out harmful advice about diets. They suggest ways to hide the disorder and generate "inspiring weight loss content" that worsens the problem.
OpenAGI released the Lux model that overtakes Google and OpenAI
Startup OpenAGI released the Lux model for computer control and claims this is a breakthrough. According to benchmarks, the model overtakes analogues from Google, OpenAI and Anthropic by a whole generation. Moreover, it works faster. About 1 second per step instead of 3 seconds for competitors. And 10 times cheaper in cost per processing 1 token.