The Bangladesh Mask Trial is re-analyzed and it falls apart
And, we are still missing an endpoint
Recently Marina Chikina, Wes Pegden, and Ben Recht published their re-analysis of the Bangladesh mask RCT, and the trial appears to fall apart. I will do my best to summarize the key issue in the study, and why this re-analysis is provocative.
First, let’s start with basics. There is a reason that the CDC, WHO and Fauci himself advised against community masking in early March 2020— and it was not to protect the supply for health care workers. The true reason is simple: the pre-existing evidence was poor. That’s the opinion of the Cochrane collaboration, and a systematic review that I participated in on the topic. These agencies truly did not believe it would help because that’s just what the evidence said.
How did COVID19 studies change the evidence? Well, there was a sea of low quality observational studies. They are not worth considering, as noise is 2 orders of magnitude larger than signal. I have debunked dozens in these pages and on YouTube.
There was one individual level randomized trial (DANMASK) that failed to find a benefit, but was limited by low power to exclude a small benefit. There were just 2 cluster RCTs run globally— and none pertaining to children.
One cluster RCT has not been published, and the other is the Bangladesh study. Bangladesh is a cluster RCT that randomized adults in villages to free masks (surgical or cloth) and encouragement to wear them or not, and followed people for COVID19 outcomes. The study found surgical masks lowered rates of COVID19 —though the effect is very small and applies only to adults pre-vaccine and pre-natural immunity and not cloth masks.
Enter the re-analysis. The authors noticed that there was a difference in the number of people enrolled in the study. It looked like ~9% more people enrolled in the free mask arm. This 9% is highly significant, i.e. a real difference.
Of course, the purpose of a randomized trial is to minimize confounding and balance outcome distributions in the absence of treatment effect, but imbalance in the size of groups suggests that something might have happened that jeopardizes this fact.
What would cause more people to sign up for the treatment arm (free mask) than control arm? One possibility is that concealment was violated, and people knew that they might get something for free in 1 arm, but did not feel they would get anything in the other arm.
If participants could see a big truck or boxes in intervention villages, but not see that in control villages, they may be more likely to enroll. In fact, 9% more likely!
This has huge implications. Is the extra 11th person in the mask arm the same as the 10 people in control arm? Or is this the type of person that only enrolls on the margin? Only enrolls if they are getting something for free, but not otherwise, and thus slightly less likely to properly report COVID symptoms (perhaps they report less or differently) and less likely to follow through with testing?
The authors argue this is possible, and this threatens the entire trial. Assuming these people are just a little different, can cause the entire trial results to tip. Their paper nicely probes this statistically and is worth your time.
I have one separate question about this study. The strongest secondary endpoint— the only one truly bias resistant— is random seroprevalance (which does not rely at all on reporting). This endpoint remains listed on ClinicalTrials.gov, but unreported. It must be completed and reported.
Finally, it is worth restating: Bangladesh has no relevance to children, or post-vaccine. It also doesn’t not apply after sero-prevalance rises. In other words, it is not relevant for 2022 America, but would be good to pin down for historical reasons.
Here is my thread on the topic
Vinay Prasad's Observations and Thoughts is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.