Even GPT-5 Failed This Human Attention Test

A decades-old psychology test exposed a surprising weakness in AI’s ability to stay focused.

A classic psychology test has revealed a surprising weakness in some of today’s most advanced artificial intelligence systems, suggesting that AI attention may work very differently from human attention.

Researchers led by Suketu Patel investigated how large language models (LLMs), the technology behind systems such as GPT-5, Claude, and Gemini, handle a well-known cognitive challenge called the Stroop task. The findings suggest that while AI can perform impressively on many complex tasks, it may struggle to maintain focus when faced with competing information over extended periods.

What Is the Stroop Task?

The Stroop task is a classic psychology experiment that has been used for decades to study attention and mental control. In the test, participants see words that name colors, such as “red” or “blue,” displayed in colored ink.

Sometimes the word and the ink color match. For example, the word “red” may appear in red ink. Other times they conflict, such as the word “red” appearing in blue ink.

Participants are asked to identify the color of the ink while ignoring the meaning of the word itself.

Although this sounds simple, it creates a mental conflict. Most people are highly practiced at reading words automatically, so suppressing that instinct requires what psychologists call executive control. This refers to the brain’s ability to focus on a goal, resist distractions, and override automatic responses.

Humans typically take a little longer to answer when the word and color do not match, a phenomenon known as the Stroop effect. However, even when the task becomes lengthy, people generally maintain high accuracy and remain focused on the instructions.

AI Performs Well at First

To see how modern AI systems would handle the same challenge, the researchers tested several leading language models using lists of color words.

When presented with short lists containing five words whose meanings conflicted with their ink colors, the models performed surprisingly well.

GPT-4o achieved 91% accuracy on these shorter tests. Claude 3.5 Sonnet also performed strongly.

At first glance, the results suggested that AI systems could successfully follow the task and ignore the distracting word meanings.

Performance Collapses as Lists Get Longer

The picture changed dramatically as the researchers increased the length of the word lists.

GPT-4o’s accuracy dropped from 91% with five words to 57% with ten words. By the time the list reached 40 words, accuracy had fallen to just 15%.

Claude 3.5 Sonnet proved more resilient, maintaining stable performance through lists of 20 words. However, it too experienced a sharp decline, falling to 24% accuracy when faced with 40 words.

The researchers observed similar patterns in GPT-5, Claude Opus 4.1, and Gemini 2.5.

Performance became even worse when matching and mismatched color words appeared together within the same list. Under those conditions, accuracy on the mismatched items dropped to nearly zero.

Why Humans and AI Respond Differently

The results point to an important difference between human cognition and the way large language models process information.

Like people, AI systems have effectively received far more training in recognizing and interpreting words than in identifying colors. This creates a natural tendency to focus on the written word.

However, humans are generally able to suppress that automatic response and stay focused on the task they have been instructed to perform, even across long sequences of items.

The language models, by contrast, increasingly reverted to reading the words instead of naming the colors as the tests continued. In other words, they appeared to lose track of the original goal.

According to the researchers, this breakdown suggests that the attention mechanisms used by transformer-based AI systems differ fundamentally from the biological attention systems found in the human brain.

A Window Into AI’s Limitations

Large language models have demonstrated remarkable abilities in writing, reasoning, coding, and conversation. Yet studies like this highlight that impressive performance does not necessarily mean AI processes information the same way humans do.

The findings suggest that modern AI may have hidden weaknesses when tasks require sustained focus, inhibition of automatic responses, and long-term maintenance of specific instructions.

As AI systems become increasingly integrated into everyday life, understanding these limitations could be just as important as measuring their strengths.

Reference: “Deficient executive control in transformer attention” by Suketu Chandrakant Patel, Hongbin Wang and Jin Fan, 2 June 2026, PNAS Nexus.
DOI: 10.1093/pnasnexus/pgag149

Never miss a breakthrough: Join the SciTechDaily newsletter.
Follow us on Google and Google News.