A new JAMA IM paper on covid shots contains a preposterous result that invalidates the entire paper

Post-publication peer review

A new paper in JAMA IM claims that the 24-25 covid booster reduces cardiovascular events, but the Supplement contains data that invalidates the entire paper. Here is the main result:

Of course, this is not a randomized trial. It is an observational study. So the first question you have to ask is: are the groups well matched— did the authors create two comparable groups whose only difference was COVID-19 vaccine receipt.

In the supplement it is clear they do not achieve adequate matching. All cause mortality curves separate by day 10. That is simply impossibly fast. The MACE separates by week 5, and the all cause death separates by week 2. The only explanation is that the no vaccine group is fundamentally different from the outset of the study. The healthy vaccinee effect. The paper failed to achieve a suitable control arm.

That’s all that needs to be said. Do not pass go, do not collect 200 dollars. The entire paper, and all included results, is flawed. Not only should it not be in the news, the paper should not be in print. It is hopelessly flawed.

Ironically, this very type of error that wasn’t detected by JAMA IM editors was pointed out by us previously in…. JAMA IM.

It is also an error we pointed out for other papers in NEJM.

On a technical note: why are negative controls undisturbed from nullity while all cause mortality is proving to be a useful falsification test? I suspect that all cause mortality may be a more versatile negative control as it captures health differences across the entire body, rather than specific domains. Additionally, power. Raw numbers were not provided for the chosen negative controls— yet likely they are also severely underpowered, and all cause mortality is not.

The media unfortunately lacks in house critical appraisal of science, and has propagated the error. The take home lesson is clear: the headlines are often incorrect.

If you support independent analyses, subscribe here.