Can you repeat that please?
Behavioral New World
August 1, 2023
Can you repeat that please?
One aspect of good decision-making involves recognizing our cognitive limitations and biases, many of which have been discussed over the past three-plus years in this newsletter. Another aspect is the quality of the evidence about the way the world works—can we rely on it to make good decisions?
Evidence starts with research by academics, scholars in think tanks, and government policymakers, among others; I’ll refer to this as “original research.” Often this knowledge is offered to a broader audience by interpreters of that research (e.g., me 😊) via books, newsletters, podcasts, and so forth.
Thus, the quality of the evidence we rely on in decision-making depends on three things: 1) the integrity of the original research;[1] 2) how accurately that research is interpreted for broader audiences; and 3) whether the (non-fraudulent) research can be replicated. Replication gives us greater confidence in results or conclusions if different researchers run similar experiments and get similar results.
This newsletter discusses the question of replication. The information below suggests that we should default to some degree of skepticism when confronted with new, not well-confirmed results. I will return to the other two issues in a future newsletter and suggest how laypersons can evaluate research from a distance.
The Reproducibility Project[2] showed that fewer than half of prominent psychology studies could be replicated. Yes, less than 50%! And of course, psychology is an important contributor to behavioral economics, the overarching theme of these newsletters. (I hasten to add that this problem plagues all disciplines, including economics for which about a third of studies do not replicate.[3])
Below I report briefly on six findings that were well publicized but which have not stood up well to replication. I include a link for each in case you want to know more. In no particular order:
Power posing, priming (the first two) and others not covered here are discussed in https://www.bps.org.uk/research-digest/ten-famous-psychology-findings-its-been-difficult-replicate
Power posing.
A well-known study and related TED talk discussed the effects of standing in a power pose—hands on hips, feet a bit wider than shoulder width. Initial research suggested that the Superwoman pose, as this stance is sometimes called, will make you feel more confident. It will also (allegedly) increase levels of testosterone (a good thing) and decrease levels of cortisol (not a good thing).
Subsequent research suggests that people standing in a power pose do feel more confident. But studies have failed to consistently find the hormonal effects of power posing, so there is doubt about the ultimate benefits of power posing.
Priming.
It is reasonable to think that our thinking is influenced by environmental factors of which we are not aware. Example: A well-known study purported to show that exposing people to words related to aging made them walk more slowly. And, as claimed, does holding a warm cup of coffee make you more generous?[4]
These two phenomena do not reliably replicate. That’s not to say that priming never works, only that these two priming techniques appear to have little or no influence.
The endowment effect. https://scholarship.law.bu.edu/faculty_scholarship/521/
The endowment effect says that if we own something (we are “endowed” with it), we put a higher value on it than we would be willing to pay for it, that is, how much we would value it if we didn’t own it. That seems a tad irrational—doesn’t the value just depend on its value to you, not whether you own it or not?
But we don’t have to answer the question because the endowment effect has been called into question.
The marshmallow test. https://anderson-review.ucla.edu/new-study-disavows-marshmallow-tests-predictive-powers/
A clever (IMHO) study showed that young children who were able to resist eating a marshmallow for a relatively long time had better life outcomes than children who couldn’t resist. The idea is intuitive—if you can defer gratification, you are likely to save more and have better health in the future, for example (see last month’s newsletter).
In spite of this seemingly reasonable conjecture, there is reason to doubt the conclusions of the marshmallow test.
The honesty pledge. https://www.pnas.org/doi/10.1073/pnas.2115397118
Original study: People who sign a pledge to be honest cheat less on subsequent tests. This phenomenon is somewhat like priming, but the environmental influence is not hidden.
And you guessed it—that conclusion is not strongly supported by follow-up studies.
The Dunning-Krueger effect (my July 2021 newsletter, alas). https://rb.gy/k3728
The Dunning-Krueger effects states that the least informed (or skilled or experienced) don’t know how little they are informed (or skilled, or experienced). Loosely speaking, one might say that they don’t know what they don’t know.
But if you click on the link above, you’ll see that this effect doesn’t hold up well (or you can just take my word for it).
Have I convinced you to default to a skeptical mindset when presented with (especially) “new and surprising” findings? It can be hard to know what to believe; I will provide some guidance in future newsletters. And note that it is probably unwise to cross the line over into cynicism--it is not good for your mental health.[5]
You can subscribe to this newsletter for free at johnhowe.substack.com
[1] In an irony that I could not have dreamt up, a recent scandal involves a researcher whose focus was dishonesty! See https://www.forbes.com/sites/christianmiller/2021/08/30/an-influential-study-of-dishonesty-was-dishonest/?sh=18692c7d2c72
[2] https://en.wikipedia.org/wiki/Reproducibility_Project
[3] https://nap.nationalacademies.org/resource/25303/Reproducibility%20in%20Economics.pdf
[4] https://www.yahoo.com/lifestyle/sorry-holding-warm-coffee-wont-080029742.html
[5] https://www.theatlantic.com/family/archive/2022/01/cynicism-modern-ancient-true-meaning/621314/#