Sometimes preprints and press releases save lives; sometimes they are propaganda; You have to use your brain. That's a big ask these days.
Reading and interpreting medical evidence is its own competency. Being a doctor is surprisingly poor preparation. The vast majority of physicians aren't trained in reading and interpreting medical evidence, and, unfortunately, are not good.
Being a reductionist scientist is suboptimal. You may forget that a mannequin in a chamber is insufficient to infer population level efficacy in the messy real world. You forget the emergent and unanticipated effects that arise as you move from theory to reality.
Stay tuned to get my thoughts on a variety of issues.
Two people can both be good at reading papers, but still not see eye to eye because of different priors. Yet, good readers often agree on the limits of evidence. Wrong denominator. Selection bias. Residual confounding. Etc. Good readers often have a mental encyclopedia of similar situations in biomedicine. I consider people like Adam Cifu, Walid Gelad, Venk Murthy, John Mandrola and others to be good readers. Even when we disagree, we agree.
There are many e.g. of poor medical interpretation during COVID-19. MMWR published a steady stream of flawed work. One IFR meta-analysis forced a single point estimate despite sky-high heterogeneity. The authors didn't understand that there was no single IFR, but it varied by setting, and thus forcing a point estimate is a fools’ errand. It was ironic that they simultaneously cast stones at others, while making rookie mistakes.
But in my mind one of the biggest early pandemic errors, which would serve as prelude for many more, was the interpretation of the RECOVERY study.
Remember back to those early days. Doctors panicked and were giving patients random drugs with no credible evidence. Hydroxychloroquine was part of a Harvard hospitals protocol. It wasn’t just Trump’s delusion, let’s not forget. Anticoagulants were given to thousands at Mt Sinai— in a shameful uncontrolled study. It was the Wild West.
The first light in the darkness was the large RCT RECOVERY. It found dexamethasone improved outcomes for hospitalized patients on O2. Interaction coefficients were significant & the results were parsimonious. It worked as O2 demand rose, and as one was later in the course of illness. The UK investigators held a press conference to disseminate the results. Many lives could be saved if we listened.
A new article in Nature Medicine describes these events, and is worth reading. It makes this point
The moment I saw the RECOVERY result, I remembered that I had heard of the trial before. The statistical plan was online, as was the protocol. I read both. Everything was clean and pre-specified.
But doctors online, including some luminaries, started to say we should wait to treat until we read the full paper.
But that point of view was just plain stupid. First, those retractions occurred DESPITE a full paper (actually 3!). Proving that waiting for a paper was not a 100% guarantee.
But more importantly: here you had a cheap drug, impartial investigators, and a statistical analysis plan and protocol online. Yes, a paper is great (it’s like a passport), but a press release and SAP and protocol is also great (it like a drivers license, credit card and SS card to boot!) Meanwhile, what we were doing was Wild West medicine based on nothing. Of course we should switch right away. I tweeted as much at the time.
And even had an explainer on when press releases were credible and when they were not.
But, many doctors not good at appraising evidence piled on.
Of course, they were wrong.
The truth is this has been the root problem with COVID policy.
The same people not good at interpreting RECOVERY are not good at interpreting data for masking toddlers or school kids; not good at interpreting data on vaccines/ myocarditis/ VITT; Not good at reading or interpreting data on kids vaccine/ booster vaccine effectiveness. And terrible at thinking about Long COVID and school closure.
They create pockets on twitter where they are united by Political Ideology and Virtue Signaling and boost each other. They use hostility to hide their incompetence. This is an ancient practice in medicine, to be fair, and is the reason why some attendings snap at students who ask probing questions that might reveal their own incompetence.
RECOVERY was an early sign that much of #medtwitter would be incapable of handing the issues to come, and that surely came to pass.
It turns out there is a price to not teaching doctors how to interpret evidence, and spending more time focusing on rewarding people to signal virtue than think virtuously. That price is catastrophically bad policy decisions. Sad to watch, worse to experience.
Wanted to update my piece with this walk down memory lane. RECOVERY was a great moment where people who did not know anything about clinical medicine were discounting the results. The error would carry forward.