What happens when doctors don't read papers well?
What happens when doctors don't read papers well?
The case of Eric Topol and how loud voices online can harm medicine
Previously, I have expressed my concern that Dr. Eric Topol is frequently quoted by news outlets discussing contemporary research, while he repeatedly demonstrates errors in reading and interpreting the underlying scientific work. This is a huge problem for the public interpretation of science, and can lead to policy errors.
Previously in this blog, I detailed at least 4 examples, and encourage you to read these links:
Here Topol promotes a misleading observational study of Paxlovid. According to the paper, the drug appears to work immediately after taking it, which would be impossible, and clearly indicates selection bias— a key error of the paper that Topol does not seem to appreciate.
Here Topol promotes a misleading observational study claiming vaccines can treat long COVID, which has essentially the same (it works too fast to be true) error.
Here Topol inaccurately summarizes a paper saying it compares people who COVID 19 to uninfected controls, when it compares people who had COVID19 and HAVE the POST COVID CONDITION to uninfected controls— sort of a big difference, and apparently to anyone who read the abstract.
And here Topol again inaccurately summarizes a paper, falsely claiming it shows that vaccines lower the risk of heart attacks and strokes, when the paper only selects people who had EHR documented COVID19. This is a huge and important difference.
Frequently making errors in interpreting and publicizing medical evidence is dangerous. Topol famously took credit for delaying the initial vaccine EUA in the fall of 2020, a delay that would have catastrophic consequences. I explain that in my recent paper on vaccines, and it is detailed in MIT technology review.
Finally, and most famously, Topol amplified Elizabeth Holmes.
Last, I checked Theranos did not “revolutionize testing.”
It would have been obvious to anyone with critical appraisal skills that frequent finger-stick blood draws may be both technically infeasible and medically undesirable.
Enter the latest topic. Recently, two people analyzed an NEJM paper on RSV vaccination among pregnant women. Topol on his substack and Dave Allely for Sensible Medicine. Mr Allely is a medical student.
There are four basic points of critical appraisal that a good reader should have noted in the NEJM paper.
A prior RSV vaccine program had been halted due to the concern for low birth weight babies. Do you see that safety signal here?
Reducing disease from RSV is good, but is that reduction translated into a global reduction in all respiratory viruses, and all hospitalizations?
Are the benefits and risks presented similarly/ fairly?
Breastfeeding maybe an important effect modifier, was that reported?
Now let's turn to the two essays. Medical student Dave Allely nails all 4 of these issues.
“Given that GSK found a safety signal of increased preterm births in their maternal RSV vaccine study, I think it is crucial to determine whether the 1% increase in absolute risk of preterm birth is real, which this study has not done.”
“All cause LRTI [lower respiratory tract infection]: There was unambiguously zero effect on all-cause LRTI:”
“This is obviously a poor way to present bar graphs, and you can often tell the difference between efficacy and safety figures by simply looking at scale.”
“the glaring omission in my mind is breastfeeding…”
As for Topol, 0/4. The entire piece reads like an advertisement for the company
“To note, in all 4 of these trials there were no safety concerns identified besides the expected acute phase reactogenic side effects as seen with the Covid mRNA/nanoparticle vaccines.”
“Why the new RSV vaccines are a BFD| The era of rational, structure-based design of vaccines that's hitting it out of the park.”
“This is the most exciting time in the history of vaccines.”
“It has been exhilarating to see all the work from many labs around the world culminate in such a rapid succession of success stories, with many more to come.”
In contrast with Mr. Allely, Topol’s commentary provides no actual appraisal. It ignores all the key issues, and reads so much like an advertisement, Topol feels compelled to end his post, “I have no relationship with J&J, GSK, Pfizer, Moderna or any vaccine manufacturer mentioned in this piece.”
The news media frequently quotes Eric Topol on a range of medical issues, but a close examination of his output reveals numerous errors in interpretation of data, and often a superficial reading. Journalists should seek content specific experts on each story, and be careful that prominent voices on social media may not be accurate, detailed or thorough.
Not everyone is good at critical appraisal of medical literature. If you aren’t good, it is never too late to work to improve ones skillset before commenting widely.
I read papers for a living, as a health researcher. Life is short and reading papers is hard and time consuming. So I use simple heuristics to determine if I’m going to study a paper or not.
Very large studies that promote drugs are a red flag. If an effect is large enough to be clear, it should work in a small study. Large drug company studies are virtually always manipulated if not outright fraudulent.
I also prefer animal studies as they are much clearer as to result.
I pretty much avoid meta studies as their conclusions are whatever their investigators want. But they are useful for their citations.
In my experience, doctors don’t read studies. They don’t have time due to managed care. And they are often incurious and unscientific.
“ Not everyone is good at critical appraisal of medical literature. If you aren’t good, it is never too late to work to improve ones skillset before commenting widely.”
Can it be improved? As a fairly recent (within 10 years) med school graduate, I can attest that there was a lot of effort made by the schools to teach critical appraisal of literature… and then Covid happened and that entire generation of young doctors trained in evidence based medicine completely fell on its face when it came to appraising literature!
Sometimes it’s not about reading skills, it’s about resisting groupthink, and I don’t know if that can be taught in, er, groups…
Gaty.substack.com