Wide confidence intervals means an intervention could still work; Is that the standard we always use in medicine?

The practical fallacy of saying "the absence of evidence is evidence of absence"

I took the liberty of plotting the largest randomized trials of masking from the Cochrane review, and, first off, as you can see, all the confidence intervals are wide, and compatible with important positive effects.

If you pool the effect of masking on reduction in respiratory virus transmission, as I did with STATA v13.0, the confidence interval is: HR, 0.87; 95% CI, 0.75 to 1.02.

You can clearly see: the confidence interval remains compatible with an important beneficial effect (0.75!). As they say: the absence of evidence is not evidence of absence. No one has proven masking doesn’t work. In fact the MAJORITY of the confidence interval is favorable. Masks work, we just need the right trial to prove it.

Sounds like what you hear on twitter, right? Here is the problem. The Figure is not masking— it is autologous transplant in breast cancer. This is a famous pooled analysis by Don Berry. Here it is again, as it appeared originally in the JCO

And guess what. The actual confidence interval in the Cochrane report for masking on confirmed COVID and influenza is: 0.72 to 1.42. Here they are side by side:

Masking’s overall CI is worse. More of it is compatible with harm. And yet, we don’t say ‘the absence of evidence is not evidence of absence’ for auto transplant for breast cancer.

We don’t say, the effect size remains compatible with mortality benefit. We say: this practice is no longer standard of care. It was barbaric and we were wrong to endorse it for years without doing the right studies.

You may argue that the bar to reject transplant is lower because its harms are greater. A mask is not like an autotransplant. That’s absolutely true! But, on the flip side, consider that the Berry analysis is for ALL CAUSE MORTALITY and not CATCHING COVID— a far more important endpoint, but for the sake of argument, let us accept this premise. The bar to reject masks should be higher. Is there another example that might help?

Consider a seminal paper. This is a sham controlled trial evaluating the efficacy of impermeable bed covers for dust mite allergies that appeared in NEJM. This paper is famously negative.

Basically the authors took people with asthma and allergy to dust mites and made them but their bedding in impermeable bed covers or sham impermeable covers. The covers are supposed to stop the mites. The question was if this would lower exacerbations. But the trial found no difference, leading many to reject this intervention.

Yet, look at the confidence intervals! The lower bound for exacerbations goes down to .60 ! That is nearly a 50% reduction!

“No one proved that rubber sheets don’t work and is it really a big deal to sleep on uncomfortable bedding? It is a minor annoyance like masks,” you might say. And yet, this trial is famously negative, and included in our book Ending Medical Reversal, as an example of a flip flop.

The truth is: very few abandoned medical practices were abandoned on the basis of evidence that excludes the possibility of some useful treatment effect. Yet, for the most part, people do not claim “it would have worked, if only” or “this doesn’t prove it failed.” For the most part, we accept the failures and move on.

Many experts are actively treating masks differently than any other medical intervention and inventing new standards to reject practices (the entire CI must be unfavorable), which have not been used in any aspect of medicine. It is hard to believe this is genuine trial interpretation and not a desire to preserve something that many people have faith in, believe in, trust in.

In God we trust, all others must bring data. If after 3 years you cannot show when and how community mask rules improve outcomes, you have to go. Just like autologous breast transplant and impermeable bed covers.