Does the kids COVID vaccine work? Does the XBB1.5 booster work? Test negative case control studies are prone to bias

Test negative case control studies are not randomized trials

A unique type of study design is used to claim that boosters reduce symptomatic COVID and that COVID vaccines prevent severe disease in kids. It's called the test negative case control design.

Proponents argue that this study design is reliable and the best we can do. The problem is we have no clue if it is reliable. Here, I will discuss 2 recent studies and their limits.

First. What is the test negative case control design? You take all people seeking care for acute respiratory viruses and put them in two buckets. People who test positive for COVID (the cases) and people who test negative (the controls)

Then you ask: how many cases got their covid vaccine, and how many controls? If you assume that cases and controls are broadly similar, you look for more vaccination in the controls than the cases as proof the vaccine averts bad COVID 19 outcomes.

Think about that. If the vaccine prevents you from getting COVID, fewer people with COVID should get it.

Its logical but is this study design reliable?

In other words, are the point estimates of vaccine effectiveness correct?

First, consider how you would validate this design. The gold standard way to validate it would be the same as in RCT DUPLICATE, a target trial validation that we discussed on my podcast.

That would mean: pick a set of test, negative case control studies, and run randomized trials on those questions, and check the concordance. This has never been done. Cherry picking examples doesn't count.

Second, in the absence of such data, one has to try to think about the method and deconstruct it. What biases might be present?

Consider the recent CDC study of kids vaccines. It looks at 7,434 kids who presented with acute respiratory infections to the ER (some were hospitalized). 387 had covid and ~7000 didn’t. 10% of case kids had the COVID shot and 15% of controls.

Even if you believe this, the odds ratio is 0.66, which means that COVID shots barely reduce the event.

But here is why you might not believe it. Control pts were not similarly sick. 56% needed oxygen, which was only true for 29% of cases.

One alternative way to explain the data is that the control kids probably are sicker/ more vulnerable. Their parents are scared and get them COVID vaccines at higher rates than the general population b/c their kids are vulnerable, and the media scared them to get the shots.

When they bring these kids into the ER, they are really sick with respiratory viruses (again b/c they are vulnerable). Test negative case control assumes these are kids drawn from the same population, and are presenting with the same condition^.. But there is no guarantee this is happening.

On a side note, the paper ironically shows that COVID is not so bad for kids. The Right column is deaths from COVID among kids with COVID (PS only 10% got the COVID shot - so that doesn’t explain the low rate). Also only 5 kids were in the hospital for more than 5 days. The risk to kids is worse from other respiratory viruses.

Now consider the recent study “proving” the XBB booster works. It looks at people who present to a Walgreens for a COVID test. (PS who does that?)

Here is what they say: (quote)

For the full analysis, case-patients were persons who received a positive nucleic acid amplification test (NAAT) result; control patients were those who received a negative NAAT result. Tests among persons fulfilling any of the following criteria were excluded from analyses: 1) self-reported immunocompromising condition^¶¶; 2) reported receipt of Novavax as the most recent dose and reported receipt of <2 total COVID-19 vaccine doses***; 3) reported receipt of a Janssen (Johnson & Johnson) COVID-19 vaccine dose after May 12, 2023^†††; 4) receipt of the most recent dose <7 days before the date of testing or during September 1–12, 2023; 5) receipt of a COVID-19 vaccine <2 months before date of testing for those who did not receive an updated COVID-19 vaccine dose; or 6) registration for testing with a version of the questionnaire that only reported month and year of the most recent vaccine dose rather than calendar date. In addition, tests from persons reporting receipt of a positive SARS-CoV-2 test result during the preceding 90 days^§§§ were excluded.

First, the exclusions are bullshit. You should not exclude pts with recent booster. You own those outcomes. Why exclude people who got COVID vaccine earlier without booster. Silly. Look at the missing data from people who report month and year only. Why exclude self reported immunocompromised status?

Honestly,. this looks like p-hacking. They ran the analysis without these exclusions and got the wrong answer, and then made the adjustments. How do I know they didn’t do that?

3,295 people had COVID+ tests and 5,927 were controls. 78% of cases got some COVID vax and only 71% of controls.

That doesn’t look good! If this is not biased, this design shows that COVID vaccines increase the rate of getting COVID! OR of getting COVID with prior vax is 1.44! Or people who got vaccine were 44% more likely to get COVID (note: I do not believe this, but these data also show this).

If you were a case, 9% got COVID vax, while this was true for 14% of controls. This translates into modest efficacy against any symptomatic disease.

But think about how this could be flawed. People who are cases probably have some similar symptoms, but control patients might not be similar. They might include people who are presenting with really vague nonspecific things. And these *worried* people might have disproportionately gotten the vaccine. This assumes that worried people don’t disproportionately present with COVID, but that’s possible. When you have real symptoms consistent with COVID both vaxd and unvaxd might behave more similarly than with nebulous symptoms. The truth is no one knows.

Test negative case control makes a little sense, but it lacks validation. There are many biases that might be present. This is not good enough for a mass vaccination campaign. If you want to boost kids and millions of Americans year over year, you need robust randomized data powered for severe disease. This design is too malleable and prone to bias.