Will Science do better post COVID19?
Will Science do better post COVID19?
The truth is we were not good long before COVID19
Yesterday, I deconstructed the paper by the CDC about whether COVID19 causes diabetes in kids. You can read that here or watch the video (my face says it all)
My colleague Venk Murthy (a very smart guy) noted that the CDC analysis and the uncritical news coverage was promoted by many experts. He writes:
His tweet got me thinking. Do online medical experts need to work to regain credibility? The more I thought about it, though, I rejected his conclusion. Medtwitter cannot regain credibility it never had.
The truth is reading journal articles & thinking about science is a skill, and most voices on twitter & many in the academy are not very good at this skill. Consider some examples. First covid19, then pre-covid19.
COVID19
MedTwitter thinks John Ioannidis is satan, and his work on IFR is negligently wrong, while higher estimates are correct. To address these concerns, John performed an umbrella review of all the systematic studies. I truly doubt more than 10 people on Medtwitter read this paper in entirety. If they did, they would begrudgingly say: John makes very good points in his appraisal, and the overlooked deficiencies in work of his critics is clear.
Masking literature
The evidence to claim that community wide cloth masking should have been pursued is abysmal. It took us many months, but we reviewed all of the available literature in a 25,000 word working paper. Recently, I saw that others have finally begun to admit the truth. But, Why did they get it wrong for so long?
MedTwitter made all the classic mistakes of medicine. Assuming that a plausible mechanism of action was sufficient to know an intervention works. Bio-plausibility is a poor guide, as both successful and unsuccessful interventions are bio-plausible. They accepted deeply confounded studies, just like the recent COVID causes diabetes study. They felt that inaction is always worse than action— another classic fallacy. They felt that randomized trials could not be performed because the intervention was a “parachute”. We have tried for years to point out that human beings are quick to believe their interventions have tremendous effect sizes, but that is seldom true. See our CMAJ paper
Evidence for restrictions in kids.
MedTwitter is completely off the mark in its appraisal of evidence for restrictions in children. The World Health Organization & UNICEF do not advise any child under 6 to be masked, and yet many American scientists believe that they ought to. They don’t appreciate that there is no evidence to support such a claim; Instead, they return to the cognitive delusions of reductionism & action bias. Recently, when they finally accepted evidence that cloth masks do not work to slow viral spread in a population; they oddly called for use of more restrictive masks kn95 or equivalent in kids. These also lack any germane randomized trials to show benefit in any context at these ages.
Vaccination policy
Throughout this pandemic, other nations have done different things around vaccines than the USA. The UK is currently advising healthy 5 to 11 year olds to get 0 doses of vaccine, and 12 to 15 year olds 1 dose. The USA is currently advising 2 and 3. These differences are due to legitimate uncertainty in available data, and yet many US doctors seem incapable of even considering the fact that the evidence is unclear. If it were certain, then all nations would march lockstep together. But they don’t. Instead, our ability to analyze this question is appears to be at the intellectual level: vax good, more better, me hungry, call ubereats now!
Pre-COVID19
Pre-covid19, many doctors believed that screening for lung cancer was supported by an RCT called NELSON. Nelson randomized smokers to lung cancer screening or not. Here you have an expensive screening strategy, that leads to surgeries and chemotherapy, and this was the result for all cause mortality. (right most column)
It is amazing how we in medicine are willing to spend billions on costly screening programs that do not extend life. (PS: if you think NLST has an all-cause mortality benefit, you will want to see the updated results by Paul Pinsky; the 2009 effect has evaporated).
Another good example of pre-covid error, occurred with biliary cancer. We ran 3 randomized trials of different drugs. All three trials were negative. But the least negative trial reached a p value of 0.09. Doctors said that was “close enough” and have adopted the therapy in the guidelines. But no one considered that: apriori any of the 3 trials could have hit 0.09. What is the probability, assuming the null, that 1 in 3 trials or more would hit 0.09 or less? — that was the key question that was overlooked. Still confusing. Listen to the episode of Plenary Session (a podcast to teach you to read papers better) where I explain it here.
During covid19, but unrelated, we saw experts don’t know what they are doing when it comes to moving the starting age for colorectal cancer screening. Here they make the classic error of putting too much stock in modeling, and assuming that more is always better. How did those COVID19 models work out?
In the last few years, there has been appetite to use Artificial intelligence to read mammograms, or augment prostate screening with MRIs. These technologies all assess how much cancer we can find. But this is deeply flawed. Not all things that look like cancer under the microscope behave the same. Some will kill you (you want to find them), and others will not (pls don’t find) or have spread already (pointless to find). These studies have failed to show these modalities are superior to what we are doing and yet paper after paper appears in top journals and conclusions accepted uncritically.
Our crisis
We have a crisis in medicine when it comes to understanding and appraising science. We do not teach this explicitly in medical schools, and it gets short shrift to mechanistic science. Our overemphasis on molecular mechanisms fuels the cognitive distortion that a reductionistic view is superior to empiricism. I wrote about that here.
When faced with a pandemic, we re-treated to all the old delusions. Bioplausibilty was sacrosanct— that’s why #maskswork! We can’t run RCTs— these are parachutes. Doing something is always better than doing nothing! More is better than less! Keep boosting, young man!! Newer is better than older. Disease bad; treatments good. Bad people (John Ioannidis) are always wrong (nevermind, that a year ago we all thought he was brilliant).
And that is why MedTwitter cannot reclaim what it never had. And that is why the headlines now read: COVID causes diabetes in kids, therefore vaccinate a healthy 6 year old unless you want them to have diabetes, you anti-vaxxer! Keep up your heretical views and we will do the only appropriate thing…. appoint you to be an expert British panel on vaccines.
If you like this post read Ending Medical Reversal, Malignant and subscribe to this newsletter. It will make you a better reader of science.
Gonna hijack your comment section VP for some blogstacks… hope you don’t mind
I think it’s important to realize just how powerful social media has been in allowing the pseudoscience of community masking to thrive. Just as there could be no Witch Hysteria in the 15th-18th centuries without the printing press, there could be no mask hysteria without Twitter and Facebook. Mark Twain famously said “A lie can travel around the world and back again while the truth is lacing up its boots”, and that was before we had social media to speed up the transmission speed of flawed ideas. [0]
I propose that in the distant future Metaverse archeologists will realize that the sudden mask craze of the early 20's originated spontaneously from three circular sources:
First was the PNAS study "Identifying airborne transmission as the dominant route for the spread of COVID-19" [1] published June 11 2020, which made the causal claim that wearing masks in New York City, Wuhan, and Italy caused cases to drop. It was tweeted about 31,000 times directly, picked up in over 400 news stories (of which those were tweeted a million times), cited 400 times, and read nearly 2,000,000 times.
Nearly every op-ed from that point forward, when invoking the holy word "Mask", would immediately follow-up with the responsorial hymn "which we know work", with this article hyperlinked as the evidence.
The article was so terrible, there was an immediate retraction request [2] authored by dozens of epidemiologists, scientists, and physicians who pointed out the obvious - correlation doesn't equal causation (note that nearly all the authors of the retraction letter believe that masks work, just that this study wasn't evidence of that). The retraction request was ignored both academically and socially, garnering a few hundred tweets before it died in silence. Perhaps they should have done what Adam Pearce would later do - recreate the regression tool so you too could make it tell you whatever you wanted by moving the arbitrary dates around [3]. I've long wondered how Wakefield made it into Lancet over two decades ago, I think I better understand now.
Second was on July 3 2020 when the ironic twitter handle “@nuanceORDEATH” provided an n of 70 tweet storm to settle the science [4] where they cited "…*SEVENTY* papers...Includes 31 from 2020 alone (!!)". This tweet became the de-facto response anyone would get when they dared question how we could have missed the science on this for over 100 years. He didn’t seem to realize having 31 papers appear essentially out of thin air should give cause for concern (Ioaniddis comes to mind: “The likelihood that a claim will hold up is inversely proportional to the initial attention that it gets from other scientists and the media. Large, fast-moving, "hot" fields, which can yield large financial payoffs, tend to have the worst records.)
For anyone who took the time to actually go through the entire list, they would have found the 70+ articles fell victim to common pseudoscience fallacy of "counting your hits and forgetting your misses".
It leaves out any study which doesn't support masking (roughly 30 by my count?), even going so far as to cite Meta studies like "The Role of Facemasks and Hand Hygiene in the Prevention of Influenza Transmission in Households: Results From a Cluster Randomised Trial; Berlin, Germany, 2009-2011," (tweet 45/71), then re-cites 5 of the 7 the studies cited by the meta which support masks, but leave off the 2 which didn't find masks worked. ("Surgical Mask to Prevent Influenza Transmission in Households: A Cluster Randomized Trial" and "Findings from a household randomized controlled trial of hand washing and face masks to reduce influenza transmission in Bangkok, Thailand" were both excluded in the 70).
The tweet storm also included 3 of Raina MacIntyre's studies from 2008, 2009, and 2020, but excluded her study 2017. Is it possible the 2017 study was excluded because it noted "Finally, a recent study examined the efficacy of cloth masks compared to medical mask and control groups, and found that cloth masks may increase the risk of infection in HCWs"? [5]
In summary, what @nuanceORDEATH did was deliver the fallacy of “countless counterfeits” packaged into a convenient tweetstorm that now anyone could “mic drop” whenever some pesky skeptic wondered how it could be possible for us to not realize masks could stop the flu for the last 100 years (never mind apparently we already had 40+ studies showing masks worked, which is odd, considering around 7 million people have died from the flu in the 21st century alone). If it was hard to point out how flawed the PNAS study from Zhang et al was to the credulous masker, now there 70 potential counterfeits to sort through.
There was one problem from @nuanceORDEATH’s 1/n collection going fully viral – he isn’t an expert. He has no PhD, MPH, or DPH. And that leads us to…
Third, on November 5, 2020 social media personality Katelyn Jetelina aka “Your Local Epidemiologist” repackaged @nuanceORDEATH’s tweetstorm into the most shared Facebook posts amongst her 380,000 followers.[6] Now a certified public health expert with a massive public following had given the validity to the full set of 70 articles, uncritically repeating the high level findings of the models such as “(masks) stopped more than 200,000 COVID-19 cases in the US by May 22, 2020.”… “reduces the dose of virus a wearer might receive, resulting in infections that are milder or even asymptomatic” (I completely forgot this used to be a viable theory)…and “(if) 80% of the population wore masks, this would do more to reduce COVID-19 spread than a strict lockdown”.
She concluded “I’m not entirely sure why the efficacy of mask use is still up for discussion. But, nonetheless, this simple, low-cost intervention has the potential for a large impact.”
But in a classic case of CYA / Defensive Medicine, she closes with the “Swiss Cheese Model” noting “…it’s important to recognize that mask wearing it’s not the cure all. It has to be combined with other public health efforts for ultimate success”, which cements the unfalsifiability of community masking – when masks are worn and cases rise, we can blame other pieces of cheese not being combined with masks. Which we know work.
Note – I enjoy the writing of YLE and am a founding supporter of hers on substack, I just think on this particular issue she has missed the mark and played an unwitting role in spreading the pseudoscience of community masking more than possibly any other individual other than Donald Trump. And to be fair YLE isn’t always credulous when it comes to reviewing, she does on occasion take a deep dive to pick apart studies. [7]
___________________________
[0] Mark Twain didn’t actually say that after all [8]
[1] https://www.pnas.org/content/117/26/14857
[2] https://metrics.stanford.edu/sites/g/files/sbiybj13936/f/files/pnas_loe_061820_v3.pdf
[3] https://roadtolarissa.com/regression-discontinuity/
[4] https://threader.app/thread/1279144399897866248
[5] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5705692/
[6] https://www.facebook.com/permalink.php?story_fbid=202002698114314&id=101805971467321
[7] https://www.facebook.com/permalink.php?story_fbid=206293624351888&id=101805971467321
[8] https://freakonomics.com/2011/04/quotes-uncovered-how-lies-travel/
Thank you for your analysis. MMWR reports have so many methodological limitations. It is appalling the CDC keeps accepting this "junk" and promoting it. Sad that the CDC has one agenda--nothing else matters besides getting vaccinated.