A new paper claims kids masking associated with more daycare: Truth or Fiction?

Lots of Re-Tweets but did Weathervanes read it?

A new paper has appeared Association of Child Masking With COVID-19–Related Closures in US Childcare Programs and finds that schools where all kids 2 and up wore a mask in May-June 2020 were 5.8% (ARR, 13% RRR) less likely to have 1 or more school closures due to COVID19 than schools that did not always have all kids 2+ wearing a mask in April 2020. The paper is hopelessly flawed. Here is why:

First, let me restate that finding. If, on a survey, daycare providers said that kids 2 and up wore a mask everyday that school was 5.8% less likely to ever close during the pandemic. This happened in 8.6% of day cares. Remember— we are talking about masking 2 year olds in May-June 2020, when it was not quite as stylish as it became by the fall, and certainly today.

The 8.6% of daycares that had 100% masked 2 year olds+ in May-June of 2020 were slightly less likely to ever close due to COVID 19. Roughly, masking kids at baseline meant that 40% of daycares closed, while if you didn’t mask all 2 year olds at baseline then 46% of daycares closed at least once. P = 0.04. That’s the big finding!

What was not significant in the analysis? Masking staff! It didn’t make the headline, but good news: babies can see faces in daycare! Daycares where staff always wore masks are no less likely to close from COVID19 —

Very little else was significant. Home vs. center based daycare, number of students/teachers, staggered arrival, symptom screening (kids & adults), temperature screening (kids/adults). 6 ft of distancing was not significant in the baseline multivariable model, but was significant if you combined first and last survey. Outdoor drop-off was significant at baseline, but not if you pooled the first and last survey. County level prevalence rates do look like they have something to do with this. The authors took this continuous variable and put it in 3 bins, but that is an analytic mistake in my opinion. They should have run it as a continuous variable by county of day care center.

What are the problems with this study?

1. Always masking; ever closing. Both the exposure and outcome are not aimed at the policy question we are debating. We want to know if the policy of mandating kids mask vs. allowing parents to choose whether they mask slows the spread of the virus. In other words, the exposure is not perfect masking, but assignment to such a mandate, knowing many kids will make errors. The outcome is also not did you ever close (yes or no), but rather, how many others were sick as a result of the policy.

   If a school had a mandatory masking policy but daycare workers found it hard to enforce— Keep in mind it is 2 year olds we are talking about— they may select some or most on the form, and be coded as “no masking.” This would mean the paper is not testing the policy q at all.

2. Effect size is small— Even if we believe the findings, it is a a few percentage points. It is so modest it proves cluster randomization is feasible and possible. 40 vs 46. That is not a parachute (99.9999 vs 0.00001).

3. It is all self reported. The person filling out the form might not be totally honest— they may oversell mask compliance, and they may later omit mention of COVID19 closures. They might fear it reflects badly on their business to say otherwise. During the study period many felt shame for getting COVID19, as if they didn’t do everything right. Some day care centers may feel shame. I think the entire fact this is a survey should lead us to put in the trash.

4. 19000 people were invited to fill the survey, just 6000 finished enough to use for data analysis. This is pathetically low (31%). The authors attempts to argue this is ok by comparing rough qualities of respondents vs. non, but the honorable thing would be to throw away the dataset. The response rate is too low for any conclusions.

5. Adult masking made no difference. Given that the USA has pushed workers in infant day care centers to mask, but the UK has not— out of concern that infants need to see faces, why is this finding not more prominent? To be honest I don’t trust it either, but it reeks of a double standard.

6. They authors keep dichotomizing or trichotomizing things that are more complicated. Consider the precise question on masking here:

Why are the authors not comparing every day to most days to some days for masking? Why not compare never to some to most to every for adult masking? They have the data. This leads me to the next point.

7. The p value is 0.04. The authors have massive analytic flexibility, they can choose how to combine the categories, or not. They can choose to look at baseline, baseline + follow up, follow up only. They decide what to put in the multivariable equation, and they barely score a win by dichotomizing kids masking, when they have data that would permit them to look for dose response. What am I to think? It strongly feels as if they selectively presenting models that fit a conclusion. The classic p-hacking.

   Consider that they do not present a multivariable model for follow up only. They also changed the categories on the follow up question; They removed the ‘most days’ category. Why do they not report a model asking if wearing mask at the end of study is associated with fewer closures? After all way more places (now 32%) are doing it at end of study! The proffered explanations made me chuckle.

8. The closure question asks if the program ever had to close because someone associated got covid, such as staff. Not whether the person got COVID in school. It could be that places that loved to toddler mask have bigger labor markets with many more day care providers to sub-in if one is out sick. And that the mask itself is a marker of the labor market. In other words, the paper cannot even comment on whether the COVID spread in school. Nor do the authors know that all places had the same closure policies (they most certainly do not), and closure might be dependent on other things entirely— like depth of substitute staffing. Rich daycares love toddler masking, and bragging about it, and can pay more to avoid closure by calling in subs even when some staff get COVID. It is a huge problem of their paper

9. (addition) This idea was given to be by Alli Krug. There is also the risk of differential attrition. At baseline people have some sense of the questions. People who experienced outbreaks after promising that they did everything right (kids were masking all the time, we swear) might be slightly more likely to not fill out the second survey. There are many other paths that this attrition can take, but it is a huge issue. The participants know the questions and can selectively drop out.

When you combine these facts with the discussion, the paper appears like propaganda. It goes on and on about the virtues of masking kids, it ignores or trivializes potential downsides, it endorses the CDC’s bizarre (anti WHO, UNICEF) policy. The truth is it proves nothing. Had it reached the opposite conclusion, it would never have been published in the literature because methods-wise it is garbage. The only reason it is so prominently covered is because it tells many what they want to believe. The only real take home message is that a randomized trial was possible in 2020. The effect size—at best— is modest. Not doing that trial is a failure of science.

So, unfollow any Twitter account who promotes this paper because are not good at reading science. And go ahead and shelf this gem alongside the CDC’s Maricopa and Pima county section. That’s the company it deserves to keep.