Methods Matter: An Analysis of NEJM's Singapore Study Claiming Benefits of Vaccines in Kids 5-11

By David Allely

David Allely is a third year medical student at Mount Sinai, New York City.  In this essay, he analyzes the recent Singapore Study in NEJM, claiming that vaccinating kids aged 5 to 11 lowers the risk of hospitalization by 83%.  Mr. Allely assesses this claim and its relevance to US children.

--Vinay Prasad

Methods Matter: An Analysis of NEJM's Singapore Study Claiming Benefits of Vaccines in Kids 5-11

By: David Allely

The New England Journal of Medicine (NEJM) is one of the most prestigious journals in the world. They regularly put out podcast episodes in which experts discuss guidelines, clinical trials, and current issues in medicine. Recently, I listened to a discussion with Eric J. Rubin, M.D., Ph.D., Lindsey R. Baden, M.D., Ashish K. Jha, M.D., M.P.H., and Stephen Morrissey, Ph.D. The discussion was titled, “A Look at the U.S. Government’s Covid-19 Strategy”. 

I will focus on one study they discussed, which was published in the Journal. The main takeaway, according to Dr. Jha, was a 82.7% reduction in hospitalization conferred by Pfizer’s vaccine. That’s an impressive result on its face, so let’s look at how they got there.

This was an observational study of children ages 5-11 in Singapore. It lasted 11 weeks. The uptake of vaccines was strong in the population, with 67.7% of the population fully vaccinated by the end of the study, and another 12% partially vaccinated. That left 20.3% of the children unvaccinated by April 8.

There are always issues that must be kept in mind for a study like this. These can be summarized into a single question: Is there any difference in the children who got vaccine versus those who didn’t? 

This question reveals why randomized controlled trials (RCTs) are the gold standard for evidence. If children that weren’t vaccinated behave in ways that make them more likely to get COVID-19, that would be a confounding variable. The goal of any study is to establish some level of causation. X decreases the chance of Y— vaccination decreases the chance of infection/hospitalization. 

RCTs are not typically used for studies establishing harms—the banal example of which is cigarette smoking and lung cancer—but they are often used for studies assessing whether interventions make people better off.  When we do rely on observational studies to support treatments, it’s good to keep in mind their inherent weakness.  With that said, let us turn to the study.

Title of the Study: “Effectiveness of BNT162b2 Vaccine against Omicron in Children 5 to 11 Years of Age”

Study Type: Observational

Timeline: January 21 through April 8, 2022. (<3 months)

Population: 255,936 children ages 5-11 in Singapore met the study criteria. Children with previous infection were not included. 

Note: The exclusion of previously infected children is a common theme in many studies in this space. This represents a huge asterisk for obvious reasons. It is impossible to apply the conclusions of this study to a population that has widespread previous infection, such as the United States. As of December 2021, 75% of U.S. children ages 0-11 had been previously infected with Covid, per CDC. It’s reasonable to assume that number is far higher now, almost 7 months later. 

Results: 

First, the overall burden of infection and hospitalization. There were 53,429 Covid infections, 5,342 (10%) of which were PCR confirmed. 288 children were hospitalized (0.5%). Among hospitalized children, only five received supplemental oxygen, four of whom were admitted to the Intensive Care Unit (ICU). Of these five children, one was unvaccinated, two were partially vaccinated, and two were fully vaccinated. Six children had Multisystem Inflammatory Syndrome in Children (MIS-C), four of whom were unvaccinated, one partially vaccinated, and one fully vaccinated. 

Zero deaths attributable to COVID were seen in this study. 

Note: The large difference between Covid infections and PCR-confirmed infections requires explanation. According to the study, PCR testing was done in symptomatic children with a positive rapid test. So roughly 10% of infected children had symptoms.

We can see here that children of this age are at low risk of severe disease. Zero deaths with over fifty thousand infections. Without more granular data it’s hard to interpret the morbidity of the 283 children that were hospitalized and did not require supplemental oxygen.  

Vaccine Efficacy: “In fully vaccinated children, we estimated vaccine effectiveness to be 36.8% (95% CI, 35.3 to 38.2) against all reported infections, 65.3% (95% CI, 62.0 to 68.3) against PCR-confirmed infections, and 82.7% (95% CI, 74.8 to 88.2) against hospitalization.”

Also: “In our secondary analysis, the vaccine effectiveness against all confirmed infections in the fully vaccinated group as compared with the unvaccinated group was 48.8% (95% CI, 46.9 to 50.8) at 7 to 14 days after the second dose of vaccine, 37.6% (95% CI, 35.7 to 39.3) at 15 to 29 days after the second dose of vaccine, 28.5% (95% CI, 26.3 to 30.7) at 30 to 59 days after the second dose of vaccine, and 25.6% (95% CI, 19.3 to 31.5) at 60 days or more after the second dose of vaccine.”

This fits well with what I would expect to see, given what we know occurs in adults. The vaccine provides longer lasting protection from severe disease, and shorter protection from infection. The waning infection protection seen in this study doesn’t obviate the reduction in hospitalization. The short length of the study makes it difficult to infer the longevity of the protection from hospitalization. In fully vaccinated kids, relative vaccine effectiveness against hospitalization was 87.8%, 84.5%, and 80.4% at 7-14, 15-29, and 30-59 days after second dose. 

One caveat worth mentioning is that it is always challenging to separate hospitalization due to COVID-19 vs. hospitalization with sars-cov-2 as a bystander.  The more rigorous endpoint of hospitalization requiring oxygen occurred too seldom for analysis, which is reassuring news for kids and parents, but this limitation for hospitalization alone, remains, and must be acknowledged. 

The way the data were analyzed warrants mention. The study looked at “person days at risk” for the subgroups. A classic RCT might take 200 kids, assign 100 to get the vaccine and 100 to get a placebo. Then you would look at infection, hospitalization, and compare the rates. In this observational study, some kids started out unvaccinated, entered the partially vaccinated group the day after their first dose, then entered the fully vaccinated group 7 days after the second dose. 

This presents a challenge in how you categorize an individual. The study authors chose to calculate the “person days at risk”. Let’s take two examples: 

1. Assume child A got the first dose on day 10, the second dose on day 34, and never got infected during the study (77 days total). This would count as 10, 30, and 37 days at risk for the unvaccinated, partially vaccinated, and fully vaccinated groups, respectively. 

2. Assume child B got infected on day 38 and had not received any vaccine. That would count as one infection and 38 days at risk for the unvaccinated group. Once the outcome has occurred, further days during the study are not counted for that individual.

Note: It’s a little odd they weren’t consistent with the transition between groups, as it reduces the person days at risk for the unvaccinated group, inflating the risk for that group.

This might seem insignificant, but I want to show you that it matters. A child was considered partially vaccinated one day after the first dose. That same child would be considered fully vaccinated 7 days after the second dose. This was done to allow time for the vaccine to take effect, so why not use the same standard for the first dose? The median time from first to second dose was 24 days. That means that 30 days go to the partially vaccinated group (24+6). What if the same standard was used for the unvaccinated group?

The total person days at risk for the groups were 5,118,468 for unvaccinated, 5,340,205 for partially vaccinated, and 7,405,066 for fully vaccinated. According to the supplementary appendix, that 1-6 day period following the first dose included 622,268 person days at risk, 1,494 confirmed cases, 211 PCR-confirmed cases, and 12 hospitalizations.

Inclusion of these days would have reduced the incidence rates calculated for the unvaccinated group. I don’t want to imply that this alone is reason to throw out the study, but it should be incorporated into your interpretation. This study was clearly designed to show efficacy from the vaccines. 

One last point to make before I summarize my thoughts. In the statistical analysis section, it says “Confidence intervals have not been adjusted for multiplicity and should not be used to infer statistical significance.” The basic concept here is that there is always risk in a study that you find some effect due to chance. If a study is going to analyze data multiple times, you must use a higher threshold for drawing conclusions based on the increased risk of finding something by chance.

If you walk by a basketball court and see someone make a half-court shot, you might think they’re good at basketball. If instead you arrived 20 minutes earlier and saw they missed 100 shots before that, you might change your mind. Adjusting for multiplicity is a way of controlling for how many shots the researchers took when they analyzed the data. So, we’re left wondering what the confidence intervals look like after adjusting for multiplicity. I couldn’t find it in the supplementary appendix. We are unable to say whether these results would be significant, given the potential for alternate unreported endpoints or analytic plans. 

Final Thoughts: This was an observational study that lasted less than three months and excluded children with previous infection. Use of Pfizer’s vaccine was associated with a modest reduction of infection and a larger reduction in hospitalization. The protection from infection rapidly waned to 25.6% by the end of the study. Protection from hospitalization decreased over time but remained high (80.4%) up to 59 days after full vaccination. None of the confidence intervals were adjusted for multiplicity. The design of the study was biased in favor of fully vaccinated individuals, with asymmetry in subgroup definition that artificially inflated vaccine efficacy. 

Pfizer’s vaccine has saved many lives, and they deserve our gratitude for the rapid development and rollout they were able to pull off. Somewhere along the way we lost the ability to include nuance in the conversation about using these vaccines in different populations. Take two extremes: For an elderly, obese man with no previous infection this has been a Godsend. For a 5-year-old child that had the real virus 2 months ago this does nothing. Yet it seems our so-called experts are content to make blanket recommendations for our entire population. 

These experts touted the efficacy against hospitalizations and claimed it gave them reason to push vaccines harder in children. They failed to mention that this study is only relevant to a vanishingly small percentage of American children. Quality of evidence matters and referencing this study as strong evidence with no caveats or breakdown of the methods seems sloppy for the hallowed halls of NEJM. 

Evidence-based medicine is on life support, look no further than our policy makers’ lack of recognition for natural immunity. When it comes to young children, families don’t seem to be buying it. Recent CDC data shows only around 3% of children under 5 have received a single dose.

Mr. Allely is a 3rd year medical student at Mt. Sinai in New York City, and writes the substack Knaves Notes