Will Science Split in Two?

Why the history of COVID19 will be whatever you want it to be

Fast forward 10 years into the future.  What will science look like regarding COVID19? 

School closures? Lockdowns? Cloth masks? Will science reach a consensus as to whether, under what circumstances, and how much (if any) these interventions helped?

Here is my bet:  we will get both answers.  You will be able to pick the paper that confirms your preexisting view.

Or put another way: I think a body of literature will coalesce and support the claim that lockdowns saved lives, and we should have done them sooner. Cloth masking provided a large benefit.  Schools should have been closed.  And simultaneously, I think there will be a body of literature that contends lockdowns mostly did not save lives.  Cloth masks mostly did not work, and closing schools was a tragedy.

If I were to bet on volume of results.  I bet the majority of papers on lockdowns/ masks will conclude they help, and the majority of papers on school will lament their closure, but for each topic, and all other “hot topics” such as vaccine mandates and others, you will find studies reaching both conclusions.

Why will this happen?  Why will we get both answers?

This will happen for the simple reason that analytical flexibility, multiplicity, and faith are rampant. Let me unpack those terms.

Analytical flexibility means that you can pick and choose how you study each of these things. Take lockdowns.  How do you define it?  What locations will you look at?  Countries, counties, cities, states?  How long after lockdown will you look for an effect?  What will your control groups look like?  What endpoints will you study?  And what will you adjust for?

If you think about the sheer numbers of choices researchers can make for each of these terms, and many more, it boggles the mind. There are millions or billions of possible analytic plans that could be proposed. By chance alone some will show lockdowns helped, and some will show the opposite.

Now think about Multiplicity.  There are tens of thousands of people with the skill set to perform this research.  In the years that follow, a chunk of those million analytic plans will actually be run by some investigators. 

Finally, faith.  Faith means simply that many scientists have already made up their mind about these issues.  These people will be motivated to trumpet an answer that confirms their view. 

If you put these things together-- analytical flexibility, multiplicity, and faith—even well intentioned people who report faithful will yield a bifurcated literature.  Selective reporting will drive the overall picture.

What does this mean? 

I think it means that the truth will be elusive and unlike to be found on social media.  Natural experiments (if discovered) may shed light, but the failure to generate large scale experimental evidence has already occurred.  We might be mired in darkness for a long time to come.