The reproducibility crisis and what it means for the NIH

Why scientists broke the social contract

When we say science doesn't reproduce, what does that mean?

It means that if you take the same experiment that's published in a top journal and you try to run it again--- the way you would take a recipe and try to bake the cake at home-- you don't get the same result— the cake flops.

How often does it happen?

A number of studies have found that this happens at least half the time, or, in some fields, even more often.

Why does it happen?

It happens because there's a culture where the people who do the daily work, often poorly paid postdocs and PhD students, are under immense pressure to generate results for their career. We don't incentivize truth, we incentivize discovery. Naturally, they may cherry-pick what data to show, or run the experiment many times and take the most favorable result, or even commit outright fraud.

What about oversight?

The senior investigator does not really care about the validity of the findings from their lab. I surmise this because if they did care they would make different people run the experiment many times, and they would try to replicate the results from their own laboratories, which they seldom do.

Is it still science if it doesn't replicate?

If a scientific finding does not replicate, it doesn't tell you anything true about the universe. It's just telling you about the idiosyncratic conditions under which it occurred, which is not knowledge, and pretty useless.

What does this mean for the social contract of Science?

Having such a large problem of reproducibility, and nearly no attempts to fix it, suggest that scientists have betrayed the social contract. They're happy to swallow up billions of dollars in grants, and have these billions increase over time, but they're not fundamentally interested in whether or not they're delivering true results.

But science still leads to cures?

This is absolutely true. Despite all the flaws in the system. There are still some true and useful discoveries. But that's not the question. The question is about whether or not each dollar can generate more true and useful results. That is almost certainly true.

Is reform needed?

It is a must