Will Science do better post COVID19?
The truth is we were not good long before COVID19
Yesterday, I deconstructed the paper by the CDC about whether COVID19 causes diabetes in kids. You can read that here or watch the video (my face says it all)
My colleague Venk Murthy (a very smart guy) noted that the CDC analysis and the uncritical news coverage was promoted by many experts. He writes:
His tweet got me thinking. Do online medical experts need to work to regain credibility? The more I thought about it, though, I rejected his conclusion. Medtwitter cannot regain credibility it never had.
The truth is reading journal articles & thinking about science is a skill, and most voices on twitter & many in the academy are not very good at this skill. Consider some examples. First covid19, then pre-covid19.
MedTwitter thinks John Ioannidis is satan, and his work on IFR is negligently wrong, while higher estimates are correct. To address these concerns, John performed an umbrella review of all the systematic studies. I truly doubt more than 10 people on Medtwitter read this paper in entirety. If they did, they would begrudgingly say: John makes very good points in his appraisal, and the overlooked deficiencies in work of his critics is clear.
The evidence to claim that community wide cloth masking should have been pursued is abysmal. It took us many months, but we reviewed all of the available literature in a 25,000 word working paper. Recently, I saw that others have finally begun to admit the truth. But, Why did they get it wrong for so long?
MedTwitter made all the classic mistakes of medicine. Assuming that a plausible mechanism of action was sufficient to know an intervention works. Bio-plausibility is a poor guide, as both successful and unsuccessful interventions are bio-plausible. They accepted deeply confounded studies, just like the recent COVID causes diabetes study. They felt that inaction is always worse than action— another classic fallacy. They felt that randomized trials could not be performed because the intervention was a “parachute”. We have tried for years to point out that human beings are quick to believe their interventions have tremendous effect sizes, but that is seldom true. See our CMAJ paper
Evidence for restrictions in kids.
MedTwitter is completely off the mark in its appraisal of evidence for restrictions in children. The World Health Organization & UNICEF do not advise any child under 6 to be masked, and yet many American scientists believe that they ought to. They don’t appreciate that there is no evidence to support such a claim; Instead, they return to the cognitive delusions of reductionism & action bias. Recently, when they finally accepted evidence that cloth masks do not work to slow viral spread in a population; they oddly called for use of more restrictive masks kn95 or equivalent in kids. These also lack any germane randomized trials to show benefit in any context at these ages.
Throughout this pandemic, other nations have done different things around vaccines than the USA. The UK is currently advising healthy 5 to 11 year olds to get 0 doses of vaccine, and 12 to 15 year olds 1 dose. The USA is currently advising 2 and 3. These differences are due to legitimate uncertainty in available data, and yet many US doctors seem incapable of even considering the fact that the evidence is unclear. If it were certain, then all nations would march lockstep together. But they don’t. Instead, our ability to analyze this question is appears to be at the intellectual level: vax good, more better, me hungry, call ubereats now!
Pre-covid19, many doctors believed that screening for lung cancer was supported by an RCT called NELSON. Nelson randomized smokers to lung cancer screening or not. Here you have an expensive screening strategy, that leads to surgeries and chemotherapy, and this was the result for all cause mortality. (right most column)
It is amazing how we in medicine are willing to spend billions on costly screening programs that do not extend life. (PS: if you think NLST has an all-cause mortality benefit, you will want to see the updated results by Paul Pinsky; the 2009 effect has evaporated).
Another good example of pre-covid error, occurred with biliary cancer. We ran 3 randomized trials of different drugs. All three trials were negative. But the least negative trial reached a p value of 0.09. Doctors said that was “close enough” and have adopted the therapy in the guidelines. But no one considered that: apriori any of the 3 trials could have hit 0.09. What is the probability, assuming the null, that 1 in 3 trials or more would hit 0.09 or less? — that was the key question that was overlooked. Still confusing. Listen to the episode of Plenary Session (a podcast to teach you to read papers better) where I explain it here.
During covid19, but unrelated, we saw experts don’t know what they are doing when it comes to moving the starting age for colorectal cancer screening. Here they make the classic error of putting too much stock in modeling, and assuming that more is always better. How did those COVID19 models work out?
In the last few years, there has been appetite to use Artificial intelligence to read mammograms, or augment prostate screening with MRIs. These technologies all assess how much cancer we can find. But this is deeply flawed. Not all things that look like cancer under the microscope behave the same. Some will kill you (you want to find them), and others will not (pls don’t find) or have spread already (pointless to find). These studies have failed to show these modalities are superior to what we are doing and yet paper after paper appears in top journals and conclusions accepted uncritically.
We have a crisis in medicine when it comes to understanding and appraising science. We do not teach this explicitly in medical schools, and it gets short shrift to mechanistic science. Our overemphasis on molecular mechanisms fuels the cognitive distortion that a reductionistic view is superior to empiricism. I wrote about that here.
When faced with a pandemic, we re-treated to all the old delusions. Bioplausibilty was sacrosanct— that’s why #maskswork! We can’t run RCTs— these are parachutes. Doing something is always better than doing nothing! More is better than less! Keep boosting, young man!! Newer is better than older. Disease bad; treatments good. Bad people (John Ioannidis) are always wrong (nevermind, that a year ago we all thought he was brilliant).
And that is why MedTwitter cannot reclaim what it never had. And that is why the headlines now read: COVID causes diabetes in kids, therefore vaccinate a healthy 6 year old unless you want them to have diabetes, you anti-vaxxer! Keep up your heretical views and we will do the only appropriate thing…. appoint you to be an expert British panel on vaccines.