Math Professor Debunks the Dunning-Kruger Effect

Weird Science Experiment Concept

David Dunning and Justin Kruger tested psychology students to see whether the least skilled were also the most unaware.

Debunking the Dunning-Kruger effect – the least skilled people know how much they don’t know, but everyone thinks they are better than average.

The Dunning-Kruger effect, which claims that the least skilled individuals consistently overestimate their abilities, has been debunked as a mathematical artifact rather than a reflection of human cognition. The majority of people can actually gauge their competence and knowledge accurately.

John Cleese, the British comedian, once summed up the idea of the Dunning–Kruger effect as, “If you are really, really stupid, then it’s impossible for you to know you are really, really stupid.” A quick search of the news brings up dozens of headlines connecting the Dunning–Kruger effect to everything from work to empathy and even to why Donald Trump was elected president.

As a math professor who teaches students to use data to make informed decisions, I am familiar with common mistakes people make when dealing with numbers. The Dunning-Kruger effect is the idea that the least skilled people overestimate their abilities more than anyone else. This sounds convincing on the surface and makes for excellent comedy. But in a recent paper, my colleagues and I suggest that the mathematical approach used to show this effect may be incorrect.

What Dunning and Kruger showed

In the 1990s, David Dunning and Justin Kruger were professors of psychology at Cornell University and wanted to test whether incompetent people were unaware of their incompetence.

What Dunning and Kruger Found

To test this, they gave 45 undergraduate students a 20-question logic test and then asked them to rate their own performance in two different ways.

First, Dunning and Kruger asked the students to estimate how many questions they got correct – a fairly straightforward assessment. Then, Dunning and Kruger asked the students to estimate how they did compared with the other students who took the test. This type of self-assessment requires students to make guesses about how others performed and is subject to a common cognitive mistake – most people consider themselves better than average.

Research shows that 93% of Americans think they are better drivers than average, 90% of teachers think they are more skilled than their peers, and this overestimation is pervasive across many skills – including logic tests. But it is mathematically impossible for most people to be better than average at a certain task.

After giving students the logic test, Dunning and Kruger divided them into four groups based on their scores. The lowest-scoring quarter of the students got, on average, 10 of the 20 questions correct. In comparison, the top-scoring quarter of students got an average of 17 questions correct. Both groups estimated they got about 14 correct. This is not terrible self-assessment by either group. The least skilled overestimated their scores by around 20 percentage points, while the top performers underestimated their scores by roughly 15 points.

The results appear more striking when looking at how students rated themselves against their peers, and here is where the better-than-average effect is on full display. The lowest-scoring students estimated that they did better than 62% of the test-takers, while the highest-scoring students thought they scored better than 68%.

By definition, being in the bottom 25% means that, at best, you will score better than 25% of people and, on average, better than just 12.5%. Estimating you did better than 62% of your peers, while only scoring better than 12.5% of them, gives a whopping 49.5 percentage-point overestimation.

The measure of how students compared themselves to others, rather than to their actual scores, is where the Dunning–Kruger effect arose. It grossly exaggerates the overestimation of the bottom 25% and seems to show, as Dunning and Kruger titled their paper, that the least skilled students were “unskilled and unaware.”

Using the protocol laid out by Dunning and Kruger, many researchers since have “confirmed” this effect in their own fields of study, leading to the sense that the Dunning–Kruger effect is intrinsic to how human brains work. For everyday people, the Dunning-Kruger effect seems true because the overly arrogant fool is a familiar and annoying stereotype.

Test Answer Bubble Sheet Scantron

When students are asked to rate their ability objectively, they do much better than when they compare themselves with their peers.

Debunking the Dunning-Kruger effect

There are three reasons Dunning and Kruger’s analysis is misleading.

The worst test-takers would also overestimate their performance the most because they are simply the furthest from getting a perfect score. Additionally, the least skilled people, like most people, assume they are better than average. Finally, the lowest scorers aren’t markedly worse at estimating their objective performance.

To establish the Dunning-Kruger effect is an artifact of research design, not human thinking, my colleagues and I showed it can be produced using randomly generated data.

First, we created 1,154 fictional people and randomly assigned them both a test score and a self-assessment ranking compared with their peers.

Then, just as Dunning and Kruger did, we divided these fake people into quarters based on their test scores. Because the self-assessment rankings were also randomly assigned a score from 1 to 100, each quarter will revert to the mean of 50. By definition, the bottom quarter will outperform only 12.5% of participants on average, but from the random assignment of self-assessment scores they will consider themselves better than 50% of test-takers. This gives an overestimation of 37.5 percentage points without any humans involved.

To prove the last point – that the least skilled can adequately judge their own skill – required a different approach.

My colleague Ed Nuhfer and his team gave students a 25-question scientific literacy test. After answering each question, the students would rate their own performance on each question as either “nailed it,” “not sure” or “no idea.”

Working with Nuhfer, we found that unskilled students are pretty good at estimating their own competence. In this study of unskilled students who scored in the bottom quarter, only 16.5% significantly overestimated their abilities. And, it turns out, 3.9% significantly underestimated their score. That means nearly 80% of unskilled students were fairly good at estimating their real ability – a far cry from the idea put forth by Dunning and Kruger that the unskilled consistently overestimate their skills.

Dunning–Kruger today

The original paper by Dunning and Kruger starts with the quote: “It is one of the essential features of incompetence that the person so inflicted is incapable of knowing that they are incompetent.” This idea has spread far and wide through both scientific literature and pop culture alike. But according to the work of my colleagues and me, the reality is that very few people are truly unskilled and unaware.

The Dunning and Kruger experiment did find a real effect – most people think they are better than average. But according to my team’s work, that is all Dunning and Kruger showed. The reality is that people have an innate ability to gauge their competence and knowledge. To claim otherwise suggests, incorrectly, that much of the population is hopelessly ignorant.

Written by Eric C. Gaze, Senior Lecturer of Mathematics, Bowdoin College.

Adapted from an article originally published in The Conversation.The Conversation

6 Comments on "Math Professor Debunks the Dunning-Kruger Effect"

  1. Dunning and Kruger asked students to estimate their overall ability. Gaze asked people how they felt they performed on a per-question basis. A key element underlying the Dunning-Kruger effect is people’s inability to assess data in aggregate. Ask someone how well they understand a subject overall, like astronomy, and they’ll probably overstate. Ask them how well they understand specifics, like how stars are formed, and they’ll probably get closer to the mark. It’s also about extrapolating ability, i.e., reading a headline and feeling that you understand the whole article. Gaze’s approach is valid but not comparable to Dunning and Kruger’s approach. They are each asking different questions, and the results should not be viewed as conflicting answers.

  2. Edward nailed it.

  3. Fixed gravity for you. | May 10, 2023 at 12:15 pm | Reply

    Brain-dead gravity physics con-artists, like general relativity acolytes thinking their vote for MOND carries weight and isn’t hilarious, may personify the complexities of a typical D-K effect accuser.

  4. “But it is mathematically impossible for most people to be better than average at a certain task.”

    9 score 51%. One scores 10%. Average 46.9%. 9 out of ten people are above average.

    It’s a variaton on ‘most people have more than the average number of legs.’ I’m no mathematician but it makes me wonder how valid the rest of the article is if such a simple wrong statement is made.

  5. “To claim otherwise suggests, incorrectly, that much of the population is hopelessly ignorant.”

    But we are doomed to be hopelessly ignorant. Of all the knowledge in the universe, we can only know an infinitesimal amount. The unknown unknowns surpass the known unknowns by an infinity which in turn is infinitely greater than the known, which is greater than what anybody knows.

  6. The guy two above me is in for a SHOCK when he learns what a “median” is! And he thinks he’s enough of a nerd to be commenting here.

Leave a Reply to Paul Dodd Cancel reply

Email address is optional. If provided, your email will not be published or shared.