Replicability Crisis, Peer Review And The Future Of Science

  • Date: 15/03/16
  • Brian Resnick, Vox

How could hundreds of peer-reviewed studies possibly be so wrong? There may be a way to explain it, and it’s shaking researchers to their cores.

Illustration by David Parkins/Nature

In 1998, psychologists found evidence of a tantalizing theory: We all have a finite mental store of energy for self-control and decision-making. Resisting temptations, or making tough decisions, saps this energy over time.

Willpower is like a muscle, the argument goes. When it’s tired, we’re less focused; we give in to temptation and make shoddy decisions that hurt us later. The original 1998 experiment used chocolate chip cookies, radishes, and an impossible quiz to elegantly illustrate this. Participants who were told to eat radishes and resist cookies gave up on the quiz faster that the people who were allowed to eat the cookies.

Over the years, the theory has been tested in hundreds of peer-reviewed studies, with countless stand-ins for the chocolate, radishes, and the quiz. Scientists have shown how diminished willpower can affect our ability to hold on to a handgrip, sap our motivation to help another in need, and even negatively impact athletic performance.

This huge body of research has helped ego depletion, as psychologists call it, and its offshoot decision fatigue, become the basis for best-selling books, TED talks, and countless life hacks. In an age where temptations and decisions pummel us at warp speed, it’s become an empowering concept. If we know how the system works, we can game it: President Obama famously doesn’t pick out his suits, for fear that it might deplete some of his decision-making capabilities.

But the whole theory of ego depletion may be on the brink of collapse.

Slate’s Daniel Engber reports on an upcoming study in the journal Perspectives on Psychological Science that found in a test with more than 2,000 participants across more than 20 labs, “a zero-effect for ego depletion: No sign that the human will works as it’s been described, or that these hundreds of studies amount to very much at all.”

How could hundreds of peer-reviewed studies possibly be so wrong? There may be a way to explain it, and it’s shaking researchers to their cores.

Every time scientists conduct an experiment, there’s a chance they’ll find a false positive. But here’s the scary thing: Psychologists are now realizing their institutions are structured so it’s more likely that false positives will make it through to publication than inconclusive results.

“We’re now learning that there’s so much bias in the published literature that the meta-analyses can’t be trusted,” Simine Vazire, a professor of psychology and the editor in chief of the journal Social Psychological and Personality Science, tells me.

This had led to a painful period of introspection for psychology, leaving researchers bewildered, even scared. What if more fundamental research findings — findings that have spurred books, self-help guides, and countless articles — don’t hold up to scrutiny? Does psychology lose its validity as a science?

Michael Inzlicht, a psychology professor at the University of Toronto, is a co-author on the forthcoming ego depletion paper. While he’s not ready to discuss it in depth (“I do not think it’s wise to talk about this until people can actually read the paper for themselves,” he tells me in an email), he did clarify that the result won’t spell the absolute death of ego depletion theory. “There would need to be a few more of these massive replication failures to support a claim like that,” he says.

But beyond the demise of the theory, for Inzlicht the results represent something greater, and sadder. He’s worked on ego depletion for most of a decade. His studies have been published in top journals. “I’m in a dark place,” he writes in a recent blog post. “Have I been chasing puffs of smoke for all these years?”

Depending on whom you ask, this moment is either a crisis for the science or a revolution to hold researchers and journals more accountable for flimsy conclusions.

For psychologists, the problem is not going to go away anytime soon. Nor are the solutions easy. But there’s a chance that this fire will be cleansing — and that the science of psychology will emerge from this period stronger, more effective, and more trustworthy.

Full story