Does science matter? White House COVID Czar pushes flawed study

Using bad research to advance your argument will backfire

Recently, I saw several credible scientists use flawed studies to push policies they favor. Here is example #1, the White House COVID Czar

Ashish K. Jha, MD, MPH @AshishKJha46

Almost nothing in medicine cuts risk of death by 96% Almost nothing Except the COVID vaccines Double boosted folks had 96% lower risk of death compared to unvaccinated If you're 50 or older and haven't gotten a vaccine in 2022 Please go get one now It may save your life

Edouard Mathieu @redouad

Here's the latest data from the US CDC on Covid vaccine effectiveness, with individuals categorized from unvaccinated (0 dose) to 2 boosters. During the last week of May, a full primary vaccination + 2 boosters meant a risk of death cut by 96% compared to unvaccinated people. https://t.co/xMezs1Ho44

The tweet links to this “analysis.”

I went all the way down the rabbit hole to find the methods, but it was threadbare. I can barely tell what the authors did. But here are at least some problems

1. The effect size of 96% RRR in mortality is too good to be true. Ashish is right, we seldom see effects this big. Ironically, the person who proved that is John Ioannidis in JAMA. Instead of amplifying the unlikely estimate, a scientist might want to approach it cautiously.

2. The graph must be for 50 and over, because only 50 and over have access to dose 2 of booster, but this is not stated anywhere.

3. There have to be deaths associated with COVID19 and not all-cause deaths, due to the raw risk sizes, but that is not stated anywhere in the Figure. I found it only in the underlying linked CDC table.

4. The study appears to have NO ADJUSTMENT at all for any differences between the people (besides age, presumably adjusting for 51 vs 71, for e.g.). But people who seek out booster #2 are much more likely to be well educated, in better health, of different socioeconomic status. Perhaps there are also differences by behavior, race, and many other variables. This analysis does nothing to correct for these important modifiers

5. These are presumably deaths from COVID mixed with those with COVID— the authors have no way to disambiguate this; worse, the testing practices among those dying might vary based on vaccination status, so ascertainment bias may be at play here. I.e. an unvaccinated person in the unit may get more swabs for the vid.

The truth is for the healthy 51 year old who had 3 doses, we don’t know if they derive additional benefit for death and serious illness from the 4th. If they had 2 or 3 doses plus Omicron, the proposition is very unlikely. The fact this study finds a large difference— a massive reduction in death from the 4th dose even compared to those who had 3, even after widespread Omicron breakthrough— should be a clue it is filled with confounding. It is bullshit. That should be enough reason for the White House COVID Czar to not tweet it.

Sadly, we live in a world where science takes a back seat to propaganda that helps your policy goal. But it is self-defeating. Trust is hard to earn and easy to lose. Watching scientists push flawed studies that happen to fit their policy goals will lead to further distrust from the public. Better to make your argument without this study.

In a future post, I will take on the other example, but here is a teaser

Infectious Disease Ethics @ID_ethics

This widely-shared pre-print on masks is either: A) Evidence that the authors failed epidemiology 101 or B) An elaborate hoax designed to reveal: - the poor quality of much covid research and - the evidence appraisal skills of those who promote poor quality research

Paul Mainwood @PaulMainwood

If it is a Sokal-type hoax (see here for the original) https://t.co/Hc6NtiK8cN then it has succeeded, at least on Twitter, where various accounts that claim or imply expertise have already recommended it as evidence, including the German Health minister.