Mask studies reach a new scientific low point
MMWR appears to no longer be in the science business
Masking is a divisive issue, however, at the end of the day, it is a scientific question. Pre-pandemic, community masking was discouraged because the pre-existing evidence was negative. This is why Fauci was critical of it in early March 2020 on 60 minutes. In the pandemic, masking has become a marker of politics. Good liberals wear them and bad conservatives don’t.
As a scientist stuck in the middle, with my colleagues J. Darrow and I. Liu, we performed an umbrella review of this topic. We found very poor quality data, insufficient to support community masking, particularly for years on end. Cloth masks had especially bad data. Data to support masking kids was absolutely absent. Worse however, was how little we learned during COVID19.
The CDC did not run a single randomized controlled trial (RCT). An individual RCT called DANMASK was negative, but this trial was powered for a 50% reduction in infection. Some thought that was asking too much (we will return to this)! A cluster RCT in Bangladesh was negative for cloth masks, and very modestly positive for surgical masks. However, further data revealed imbalances in the starting size— likely because the trial failed to achieve concealment, leading more people to sign up in the intervention arm (who may be less committed to report + Covid symptoms; biasing results). Furthermore, absolute event differences were v small. These facts, combined with prior literature, suggest that confidence even in surgical masks is extremely low. Meanwhile, there are no RCTs in kids, a catastrophic research failure.
Enter a new study in MMWR— the CDC’s pet journal; It is getting widely tweeted and cited, and that is unfortunate. The paper is entirely, irredeemably flawed. Its flaws are so evident that it should not have been published nor promoted. When an issue is deeply polarizing, publishing bad science helps no one. It cannot convince skeptics, proponents don’t need convincing, and it deepens mistrust in institutions. Let’s consider the paper.
The paper is a case control, test negative study. Basically, anyone in California btw Feb and Dec 2021 who got a COVID19 test could be enrolled. People were told if they tested + or -. Two days later, someone tried to call COVID-positive people. 13.4% of people who tested positive answered the phone. For each positive person, a negative person matched by age and sex who was called was identified. Only 8.9% of people who tested negative answered the phone.
We could stop right now. Very few people answered the phone. Moreover, there may be a big difference in who answered the phone. A person sick enough to be tested for COVID, who was +, might not be doing so well 2 days later. Which ones answered? How about those who tested negative? Are thee comparable people? Sadly, researchers continued….
Among people who answered the phone, people who said they had not spent time in indoor public settings were discarded. People who had a known COVID19 contact were discarded. The analysis just compares those who tested + vs - if they said they spent time in indoor public settings.
It turns out that people who tested positive were different than those who tested negative.
77% of people who tested positive got tested because of symptoms
17% who tested negative got tested because of symptoms
More people who tested negative, got tested “just cuz”
Ok, this was opportunity #2 to stop the voyage of the titanic. The reason for testing is very different. Very likely different groups of people are getting tested. People who are sick vs. people who are worried well/ people who work at companies that have been bamboozled by COLOR to offer asymptomatic testing. Sadly, the paper continued….
You know the rest of the story.
Among those who tested +, 9.3% never work mask in indoor settings and 93% wore it any time, and 60% wore it all the time. Among those who tested negative; 3% never wore, it 96% wore it any time, and 69% wore it always. Here it is
Do the math and people who tested + were less likely to wear masks than those who tested -.
The effect size is large. So large that DANMASK wasn’t underpowered after all! ;) Can’t have it both ways, can you.
The paper did one more analysis, in a subset of people— though I cannot find just how many met this inclusion criteria by date (can anyone help me in comments?)- looking at type of mask.
They end with the now viral figure (no pun intended);
Of course, they forget to say cloth mask was NOT SIGNIFICANT p = .1.
What is wrong with this paper?
The people who test + and those who test - were seeking testing for totally different reasons. You are comparing sick people to people who wanted to get tested ‘just cuz’ or whose employer was swindled by these testing companies. Some people (more who test -) were getting tested before a medical procedure— these people may be extremely fastidious pre-procedure. I know I absolutely would not want to get sick before an elective surgery, and may even further alter my behavior. (though I am pretty chill)
In case you don’t see how this is a bias, the explanation is that the people getting tested who test neg. are far more affluent/ neurotic/ or precautious and will do anything they are told more than the avg. Joe. They definitely wear masks more. Even if masks did nothing, there would be an association. If you went on twitter for months and told them carrying a Fauci statue in their pocket would protect them from COVID, you could prove Fauci statues work with this design. Put more technically, the root virtue of test negative strategy (comparability) is violated and unmeasured confounding is injected. Paper cannot be salvaged.
The response rate is poor. I thought that awful day care study had a low response rate, but this one is lower. When you get a response rate this low you wonder if you are inserting biases you cannot even imagine. Who are the sick COVID people who answer the phone. Are they the least sick ones? The sick ones don’t take calls. Who are the people who answer the phone who test negative? The most anxious ones? The most gullible? ( I never answer unknown callers). Perhaps the ones most likely to carry Fauci statues in their pocket? (aka cloth masks)
It’s self reported. Mask usage is self reported. A person who was just told they have covid19 might be more likely to assume or believe they must have slipped in terms of mask usage. Self reporting after results are known is a huge bias. A true believer will convince themselves they didn’t wear masks as well as they ought to.
The effect size is implausibly large. In the randomized trial in Bangladesh, the arm that reached significance had a 11% relative risk reduction. In this study the same mask had 66% lower odds. That is implausibly large, and should raise red flags.
This is protection to the individual. If masks worked this well, they work this well for the wearer. So you don’t need me to mask. Your surgical mask is so massively protective you that mandating I wear a mask is not needed. This runs counter to the rhetoric in this space.
As Wes Pegden points out the CDC previously ran a similar (not the same) version of this study, but it did not give the answer they wanted, so crickets
The CDC did the same study in 2020, and found a near-perfect lack of correlation, and they buried the results in a table they ignored in the actual text of this MMWR: cdc.gov/mmwr/volumes/6… I wonder how many times they've done it since then before getting the result they wanted?
By now on this fine Friday you have all seen this terrible MMWR study. It has many flaws, but I particularly am incensed that it insults my favorite observational design, the test-negative control design. https://t.co/gcvOf12kQt
Ben Recht @beenwrektThe study only examined people who had spent time in a public space AND did not have a known sars-cov-2 contact. But the analysis could have altered either of these rules. How many other analyses could the authors have attempted? The potential for multiple analytic plans is possible. Moreover, the authors may have worsen confounding with their design. The types of people tested for a company who did not have known contacts might further select for the most risk averse/ cautious types. The types of people sick, but no one they know was sick, might further select for individuals in communities less concerned with COVID19, or reluctant to tell others about a + diagnosis.
The White House is asking Spotify to censure Joe Rogan. Joe Rogan should ask the CDC to censure MMWR. The truth is the CDC has put out so many papers that borderline propaganda, that they create the space for people to seek alternative information. They have lost trust.
But I am more disappointed in smart scientists who share this essay. They are losing their credibility. I am sad to see it.
Ultimately, the CDC & NIH failed us. The agencies should have run a half dozen masking cluster RCTs under different conditions, and for different ages. We were starving, and we needed this loaf of bread. Instead, the CDC published flawed study after flawed study. It didn’t even give us crumbs; it gave us a fistful of sand. Starving, we swallowed each grain, and begged for more. Medical leaders told us to fill our bowl before it runs out. Science lies on its deathbed.
ADDENDUM
A wise person pointed out 2 more things to me.
8. the exclusion that if you encountered someone with sars-cov-2 you get pushed out of this dataset. This person writes, “Wouldn’t it make more sense to have this type of study after a positive contact..” It is a good point. Why are we making that exclusion? Masks work but not if you met someone who FOR SURE had covid? Seems a very odd analytic choice.
9. The difference in rates of vaccination are massive; yes the authors adjust for this, but it clearly reveals the drastically different sorts of people who are here. Moreover, some people are getting 1 of J&J and others 3 of mRNA, yet they don’t adjust for this. The group with negative tests may have been pumped with more vax, and residual vax effect might be being detected. Finally, do people who get vax behave the same as people who don’t. I doubt it. In short, the study is not doing well.
Thank you for this! And for fighting for kids.
One has to ask.....if masks work so well, why did places like NYC have another huge spike in cases where the population is obsessed with wearing masks.
And you would think that controlled facilities like nursing homes would have nearly zero cases. Employees are masked full time while mingling with patients and most of these facilities did not allow guests.