The Tragicomic Exasperations of Expertise

The Dunning-Kruger effect is one of those cleanly stated insights that can at once make you feel relieved and hopeless. It is a cognitive bias which lends confidence to ignorance. Wikipedia compactly describes the effect as follows:

“…people reach erroneous conclusions and make unfortunate choices but their incompetence robs them of the metacognitive ability to realize it.” They therefore suffer an illusory superiority, rating their own ability as above average. This leads to a perverse result where people with less competence will rate their ability more highly than people with relatively more competence.

This dry, academic version actually understates both the richness and emotional complexity of what is going on. This richness begins with the subjective consequences of the impasse: the expert is exasperated, while the novice actually feels contemptuous and superior. The situation is stable: the expert gropes for a way to demonstrate the validity of his view at a level the novice can understand and is reduced to sputtering incoherence, which only serves to strengthen the novice’s illusory sense of superiority. Play out the broader effects of this little piece of sketch comedy, and you get all the pathos and pageantry of human society at the grandest scales.

The Impotent Rage of Angels

Let me first broaden the matter beyond Dunning-Kruger. The academically demonstrable version is actually not the most philosophically interesting one. In this behavioral-economics form, it is a verifiable conclusion. When the novices are systematically educated, they see the error of their previous position, which in a sense is the experimental proof of the assertion that the expert is an expert. Without such proof, the D-K effect would merely be a piece of scholarly petulance.

But the broader effect is far more interesting. It is the insight behind the proverb, fools rush in where angels fear to tread. It is the resignation behind the observation, how do you explain color to a blind man and the metaphor of the frog in the well. It is the paradox underlying Plato’s allegory of the cave. My favorite example though, is from Antoine de Saint-Exupery’s The Little Prince. If you have not read that pensive little parable about a little Prince traveling from planet to planet and learning the lessons of adulthood and maturity, do so today. In it is a curious character, the Rose, who lives securely within her own little bubble of self-assurance, defenseless and arrogant, convinced that her four little thorns are enough to protect her from the universe. The Prince, who loves the rose, is left feeling at once sad and powerless to express his thoughts.

The model of proof in the Dunning-Kruger effect is based on education. But that’s where problems arise, as this little verse, variously attributed to Confucian, Arabic, Persian or Sanskrit sources points out:

He who knows not and knows not that he knows not is a fool; avoid him.
He who knows not and knows that he knows not is a student; teach him.
He who knows and knows not that he knows is asleep; wake him.
He who knows and knows that he knows is a wise man; follow him.

(aside: this sounds like the world’s first 2×2 quadrant diagram!)

If there were always an easy way to first demonstrate to the novice that he is mistaken, and then school him (suitably chastened and properly respectful) there would be no problem. Unfortunately there isn’t always an easy way to do that, and it is not just the expert’s desperation to be acknowledged or the novice’s impenetrable barrier of contempt that get in the way. There just might be no path to persuasion. Even worse, the novice may never suffer the consequences of his delusions and provide the expert with an “I told you so!” moment (or he might suffer the consequences, but rationalize them in ways that trap him even more firmly in the prison of his own delusions). And worst of all, most other people will side with the novice because the expert’s peers are too thin on the ground to come to his aid.

The Expert’s Dilemma

What makes this a true dilemma is that there is no objective way to separate this effect from other effects where the novice’s contempt for the expert is justified; experts, both self-styled and real, are rightly held in contempt for strutting around with their own halos of illusory superiority.  I’ll cover this in another post, but let’s stick to cases where the expert is “right” in some broad sense, even when there is no moment of redemption or objective proof. Let’s also keep the definition of “expert” broad. The sort of expertise that finds external validation through degrees and certifications is actually the least interesting kind. Institutionalization can kill the D-K effect entirely. When a profession, like medicine, successfully legitimizes its own inscrutable definitions of expert opinion in the eyes of the average novice, the few contemptuous novices who remain are forced to retreat and form marginal mutual-admiration cliques.

It is interesting to note, at this level, that expertise and priesthood cannot be distinguished, because the standards of truth are relative to the community that arrogates to itself the right to define it. Those of us who believe in falsifiability must ultimately acknowledge that our standards are still internal to the communities that accept them, and that ultimately we have no real weapons of persuasion against those whose truths are determined by the standards of religion.

A strange journey begins when the exasperated expert first manages to suppress his frustrated desire for validation from those he has left behind, and ask, do I actually need this person to understand? That question is pregnant with possibilities, especially when the expert  is that pioneer who lacks even the smaller pleasures of validation from greater masters and medal-awarding societies. At the risk of reducing a complex and branching psychological process to a path of linear ascent, let me propose four stages in the journey: resignation, manipulation, benevolence and doubt.

Resignation

Dickens paints a beautiful portrait of the beginnings of this journey in David Copperfield (1850), when David first comes to terms with the limitations of Dora, the child-wife who cannot follow him on his intellectual journeys. As he moves from earnest and deluded attempts to “improve” Dora, to deep marital loneliness and finally to a resigned acceptance, it is David who gains self-awareness, rather than Dora. It is perhaps one of the deeper flaws of the novel that Dickens was unable to resist inserting a moment of redemption for David:  on her deathbed, in a brief moment of adulthood, Dora drops the little-girl act and reveals her own awareness of the gap between them. Dickens’ own relationship with his first love, Maria Beadnell (believed to be the model for Dora),  apparently had no such moment. It took Great Expectations, published ten years later, in 1860, to get Dickens past his humanist weaknesses and to the unsparing eye we look for in a great novelist.

Resignation is the necessary first step. The expert’s first-order agenda is always let me educate you to appreciate me, though he will usually deny it. This has no happy ending for anybody. It arrests development in the expert, trapped by his own over-the-shoulder glance of longing. It breeds nothing but a confused mix of contempt and resentment in the novice. The real world rarely throws up truly pure cases of Dunning-Kruger delusions; more often, the false confidence and contempt for the expert in the novice is only partly genuine. Somewhere, there is usually also an element of reaction formation against a subconsciously perceived threat to self-esteem. Resignation is the acknowledgment and abandonment of this lose-lose agenda on the part of the expert.

Manipulation

Of the many possibilities in the do I need this person to understand me question, perhaps the easiest one to understand is the path to cruelty. When the ignorant are unable to interpret the consequences of their beliefs accurately, and can be relied upon to manufacture self-serving delusions to explain away failures of their world-views, they are open to manipulation by experts with clearer perceptions of the relevant realities. My favorite example of this effect is the benign manipulation practiced by Hercule Poirot in Agatha Christie’s novels. By exaggerating his foreignness, comicality and (privately ironic and self-aware) pomposity, Poirot lets the xenophobic prejudices of the other characters, and their illusory sense of superiority, lead them into careless revelations. Of course, there is invariably the perspicacious character or two who sees through Poirot’s facade and calls him out on it. Those are the delightful moments of redemption for Poirot that makes the novels vicariously satisfying (Christie can be forgiven sins of pandering that Dickens must be held accountable for, of course).

But there are less benign forms of manipulation, and each functions rather like a quiet magic trick shorn of its theatricality and acknowledgment of the agenda of deception. But ultimately, manipulative behaviors (and certain kinds of retreat into dark humor) arrest development as completely as self-serving attempts at “education.” Each is the backward-looking validation-seeking of those unable to bear the loneliness that is the consequence of knowledge. Manipulation is a more advanced behavior purely because it is more self-aware, and does not operate under a denial of egocentricity. Each meets its comeuppance: the manipulator ultimately falls prey to the nearly-as-powerful levers of manipulation available to the ignorant. The cat and the child can manipulate the adult, even if neither has a complete appreciation of the power being wielded. The fall of the falsely-benevolent egotist is no less inevitable. Tolstoi’s Father Sergius is possibly the best literary exploration of the effect.

Benevolence

But is there true, non-egocentric benefaction that the expert can provide to a contemptuous and ungrateful novice? The sort that Buddhists personify in the notion of the Bodhisattva, the enlightened one who delays his own exit, cosmic stage-left, in order to help others along? There are two interesting stories that speak to this.

The first is from the Buddhist canon itself, one of those Zen parables that I recall reading during my infatuation with the philosophy a decade ago. A pregnant teen, possibly fearful of revealing the true identity of whatever medieval Chinese-bad-ass was responsible for her condition, points to the local wise Zen monk as the father. The monk, predictably detached, silently ignores the accusation, not bothering to defend himself. The villagers dump the child at his doorstep, and ostracize him. Still he says nothing, but raises the child. Eventually, the penitent girl confesses to her lie and the villagers come to the monk, seeking forgiveness. Still he says nothing.

My second example is from Somerset Maugham’s Mr. Know-it-All. The story features a know-it-all braggart, Sam Kelada, on a voyage. Everyone on the ship, including the unidentified narrator, dislikes him. The story turns (I won’t spoil it for you by revealing the details) on the fact that Kelada is in fact truly knowledgeable and, beneath his boorish exterior, benevolent. At a moment in the story when he could have proved his expertise, he chooses quietly to hide the truth he knows and accepts opprobrium to save a woman from the wrath of her self-important, cuckolded tycoon husband.

Both stories are interesting, and yield much to analysis, but something about them bothers me: it is the idea that the consolations of benevolent expertise lie in a sense of inner certainty of greater truths than the novice has access to. To me the point of the Dunning-Kruger effect is that it suggests a much more philosophically satisfying equation between expertise and doubt.

Doubt

The mark of the expert, in my mind is the greater capacity for doubt, and the greater courage he must exhibit doing what the novice does without thought. Fools may rush in where angels fear to tread, but angels, in turn, must make leaps of faith that fools cannot even comprehend.

This angel is that strange creature, trudging through the darker landscapes of the mind, marking his own demanding milestones and standards of truth. These are not the brightly-lit journeys of rock stars seeking newer musical truths, where adoring fans can easily see the vast gaps that separate them from their idols. Nor is this the journey of the great mathematician who finds solace in rendering speechless the sole eminence grise who can understand his magnificent proof of an obscure theorem. This is the lonely one in Nietzsche’s famous passage, On the Way of the Creator, in Thus Spake Zarathustra, that I have quoted before:

Lonely one, you are going the way to yourself. And your way leads past yourself and your seven devils. You will be a heretic to yourself and a witch and soothsayer and fool and doubter and unholy one and a villain. You must wish to consume yourself in your own flame: how could you wish to become new until you had first become ashes?