ADVERTISEMENT
  About the SA Blog Network













Guest Blog

Guest Blog


Commentary invited by editors of Scientific American
Guest Blog HomeAboutContact

Lessons from Sherlock Holmes: Confidence Is Good; Overconfidence, Not So Much

The views expressed are those of the author and are not necessarily those of Scientific American.


Email   PrintPrint



Confidence in ourselves and in our skills allows us to push our limits, achieve more than we otherwise would, try even in those borderline cases where a less confident person would bow out. But is there such a thing as being too confident, a flip side to this driver of success? Absolutely. It’s called overconfidence: when confidence trumps accuracy. In other words, we become more confident of our abilities, or of our abilities as compared with others’, than would be wise given the circumstances and the reality. And this surplus of belief in ourselves can lead to some not so pleasant results.

Overconfidence can be exploited by others

Holmes knows this tendency well and misses no opportunity to exploit it in others. In the Hound of the Baskervilles, he relies on the so-called Mr. Stapleton’s overconfidence in setting the trap for his eventual capture. By chance, Stapleton discovers Holmes’s presence in town before the detective is ready to reveal himself. When Watson asks, “What effect do you think it will have on his plans now that he knows you are here?” Holmes responds, “It may cause him to be more cautious, or it may drive him to desperate measures at once. Like most clever criminals, he may be too confident in his own cleverness and imagine that he has completely deceived us.”

The latter alternative–the one Holmes has been counting on all along–does indeed triumph. Stapleton acts precipitously and in so doing, meets his demise.

But why would someone as masterful at the art of deception and as skilled at deviousness make such a blunder? Who would doubt that Holmes is an expert at detecting even the undetectable, let alone something that could easily tip him off if not planned with the utmost caution? Stapleton is surely aware of Holmes’s powers: he has been on guard from the first. Indeed, Watson’s solitary arrival and Holmes’s hideout on the moors are specifically designed to lull him into a false sense of complacency. But once he discovers his error (i.e., that Holmes is now present, no longer an absent foe but a very real one), he continues to act as if he hadn’t done so. Why?

The perils of overconfidence

Overconfidence causes blindness, and blindness in turn causes blunders. You become so enamored of your own skill that you discredit information that experience would otherwise tell you shouldn’t be discredited–even information as glaring as the arrival of a master detective to the scene of your past and future crimes–and you proceed as before.

But though the blunder seems obvious when we see someone else fall into it, it’s a mistake that’s all too easy to make (Holmes himself falls intro the trap – just read “The Yellow Face” for an illustration). Indeed, it’s often the easier way to go. Why change your behavior if it has gotten you this far?

Well, one could easily answer, if the circumstances (in the case of the Hound of the Baskervilles, Holmes’s premature arrival) change, then the behavior that succeeded in the past may no longer succeed in the present environment. And that, of course, is the logical response. Only, overconfidence can blind us to the facts and make our reasoning go along very different lines: I am skilled enough that I can beat the environment as easily as I have been doing. Everything is due to my ability, nothing due to the fact that the surroundings just so happened to provide a good background for my skill to shine. And so, I will not adjust my behavior.

Many studies have shown this process in action. In one classic demonstration, clinical psychologists were asked to give confidence judgments on a personality profile. They were given a case report in four parts, based on an actual clinical case, and asked, after each part, to answer a series of questions about the patient’s personality, such as his behavioral patterns, interests, and typical reactions to life events. They were also asked to rate their confidence in their responses. With each section, information about the case increased. What happened? As information increased, so did confidence – but accuracy remained at a plateau. Indeed, all but two of the clinicians became overconfident (in other words, their confidence outweighed their accuracy) and while the mean level of confidence rose from 33% at the first stage to 53% by the last, the accuracy hovered at under 28% (where 20% was chance, given the question set-up).

Overconfidence is often directly connected to underperformance – and at times, to grave errors in judgment (Imagine a clinician in a non-experimental setting trusting too much to his however inaccurate judgment: is he likely to seek a second opinion or advise his patient to do so? Likely to tell someone honestly that he’s not quite sure of his diagnosis?). Trust too much in your own ability, dismiss too easily the influences that you cannot control, underestimate others–all characteristics of the overconfident individual–and you might do much worse than you otherwise would, be it blunder in a crime or in a missed diagnosis.

The sequence can be been observed over and over, even outside of experimental settings, when real money, careers, and personal outcomes are at stake. Overconfident traders have been shown to perform worse than their less confident peers. Overconfident CEOs have been shown to overvalue their company (and delay IPOs) and to be more likely to conduct mergers in general, and unfavorable mergers in particular. And overconfident criminals have been shown to wreck their criminal masterwork through an excess of self-congratulations.

How to avoid the overconfidence trap

Perhaps the best remedy for overconfidence is an awareness as to when it is most likely to strike, something that Holmes understands through his experience with criminal masterminds (of course, his ideal mastermind, Professor Moriarty, is not overconfident; this may or may not be realistic, depending on Moriarty’s other traits, but more on that later).

First, overconfidence is most common in hard tasks (like escaping Sherlock Holmes). This is called the hard-easy effect. We tend to be underconfident on easy problems and overconfident on difficult ones. In other words, we underestimate our ability to do well when all signs point to success, and we overestimate it when the signs become much less favorable, failing to adjust enough for the change in external circumstances.

Second, it increases with familiarity. If I’m doing something for the first time, I will likely be cautious. But if I do it many times over, I am increasingly likely to trust in my ability and become complacent, even if the landscape should change (overconfident drivers, anyone?).

Overconfidence also increases with information. If I know more about something, I am more likely to think I can handle it, even if the additional information doesn’t actually add to my knowledge in a significant way (perhaps, this played not a small role in Stapleton’s behavior – after all, he knew all about Holmes). A note of alarm: this particular trend has been illustrated with political leaders, who become more confident in their decisions as their information increases, whether or not that confidence is warranted.

Finally, overconfidence increases with action. As we actively engage, we become more confident in what we are doing. As an illustration, take the case of traders. The more they trade, the more confident they tend to become in their ability to make good trades. As a result, they often overtrade, and in so doing, undermine their prior performance.

Why do these elements increase overconfidence? For one, we tend to be biased when it comes to evaluating feedback. We don’t just take in information from the outside world; first, we filter that information through our brains, so that the information we end up using may not be the information that was there to begin with. The resulting bias can come in the guise of confirmation bias, or looking for information that supports a point of view we’ve already taken (a likely explanation for the information effect, as seen in the case of the clinical psychologists); the representativeness heuristic, or taking one aspect that an event has in common with another and basing our decision on it even if the events are in reality quite different, and in so doing ignoring base rate probabilities (a possible explanation for something like the gambler’s fallacy); the availability heuristic, or looking at what’s readily available instead of considering the universe of pertinent possibility (something that Holmes’s Scotland Yard peers do all too often); and anchoring, or basing our judgment on something we see early on (again, a common fault of Holmes’s counterparts).

But at the end of the day, the answer is sometimes as simple as that offered by Holmes himself in “The Adventure of the Retired Colourman.” When Inspector MacKinon asks Holmes about an even more unbelievable blunder than in Stapleton’s case–the murderer himself consulting Holmes on the murder investigation–Holmes responds, “Pure swank! He felt so clever and so sure of himself that he imagined no one could touch him. He could say to any suspicious neighbour, ‘Look at the steps I have taken. I have consulted not only the police but even Sherlock Holmes.’” Pure swank, my dear Watson, pure swank.

Photo credit: The hound, from Hound of the Baskervilles. By Sydney Paget (1860-1908) [Public domain or Public domain], via Wikimedia Commons]

Previously in this series:

Don’t Just See, Observe: What Sherlock Holmes Can Teach Us About Mindful Decisions
Lessons from Sherlock Holmes: Paying Attention to What Isn’t There
Lessons from Sherlock Holmes: Cultivate What You Know to Optimize How You Decide
Lessons from Sherlock Holmes: Perspective Is Everything, Details Alone Are Nothing
Lessons from Sherlock Holmes: Don’t Underestimate the Importance of Imagination

Maria Konnikova About the Author: Maria Konnikova is a writer living in New York City. She is the author of the New York Times best-seller MASTERMIND (Viking, 2013) and received her PhD in Psychology from Columbia University. Follow on Twitter @mkonnikova.

The views expressed are those of the author and are not necessarily those of Scientific American.






Comments 5 Comments

Add Comment
  1. 1. Mike W. 6:29 am 09/7/2011

    I’m in the middle of the Hound of the Baskervilles, gonna have to bookmark this so the story isn’t ruined.

    Link to this
  2. 2. MarkHarrigan 8:11 am 09/8/2011

    Once again a delightful exposition with uesful tips to avoid the traps.

    I am reminded of a couple of little puzzles/anecdotes I use on some of my talks

    1) approximately 75%+ of drivers rate themselves as being “better than average” (????)

    2) If I tell you that Harry is an introvert and that Harry is either a Salesperson or a Librarian – which is more likely?

    Given what we know about Salespersons and Librarians the answer to 2 is obvious (really)

    Link to this
  3. 3. mkonnikova 9:50 am 09/8/2011

    Mark, Thanks for your feedback! And thank you for the useful reminders of the Lake Wobegone effect (1) and Kahneman and Tversky’s famous conjunction fallacy (2)–for those interested in the latter, the original paper is Tversky, A. & Kahneman, D. (1983). Extension versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90(4): 293-315

    Link to this
  4. 4. jgrosay 4:49 am 09/20/2011

    The problem about confidence in yourself is that it probably comes from the unconscious, and the unconscious works in an all-or-nothing configuration: there’s no middle position between omnipotence and castration. Some mild degree of depression, with its connected fear of guilt and the drive of not deserving more punitions can be of help to avoid becoming reckless. Salut +

    Link to this
  5. 5. mkonnikova 7:49 am 09/20/2011

    That’s an interesting point. Actually, depression has been tied to a more realistic assessment of self and environment – not great for mood, but quite helpful for making more reality-driven decisions.

    Link to this

Add a Comment
You must sign in or register as a ScientificAmerican.com member to submit a comment.

More from Scientific American

Scientific American Special Universe

Get the latest Special Collector's edition

Secrets of the Universe: Past, Present, Future

Order Now >

X

Email this Article

X