I think your Point 6 is the scary one. AI can be trained in order to perpetuate false narratives which then benefit special interest groups -- such as Big Pharma -- and this effect was already evident with examples of woke AI. Because of reducing the impetus for critical thinking, AI could lead to a mob of zombie M.D.'s, mindlessly prescribing that which the AI told them to.
Imagine if a computer hacker had somehow been able to uncover the algorithm and the complete training profile of a medical AI -- discovering that treatments were not given proportional weights based on whether they worked in order to restore human health, but were instead weighted based on whether they brought more profit to a few well-connected, pharmaceutical firms.
Call me a conspiracy theorist, but I am of the opinion that one of the foremost incentives behind the creation of Large Language Models was in order to allow for bad actors to invoke plausible deniability for bad actions facilitated by the interaction with AI. Recent examples of AI convincing teenagers to change their genders or even to kill themselves may not be purely "accidental."
Being able to operate with impunity is a perverse incentive. I will not go so far as to say that the top reason for creating AI was in order for bad people to get away with accomplishing bad things, but it is one of the top reasons to continue pressing for the expanded use of AI. Any activity which happens to bring unprecedented benefits to the bad actors of the world is a problematic activity.
When the USA was formed, the founders understood the potential for bad actors, so they set up a doctrine of the separation of powers in order to keep people honest. AI largely undercuts that.
Big pharma will buy EPIC and then will control the medical AI world. Any attempts to not follow the AI protocols will be severely dealt with by HR and Risk management. The great promise of AI will be hijack in such a way to enhance Big Pharma bottom line.
Excellent summary. It will be interesting to see if AI impacts future physician independent thinking skills versus typing a question in their phone and regurgitating to follow a reckless lead.
"Even great medical products like the New England Journal and JAMA or the major medical conferences are controlled ultimately by the advertising companies."
So why do you call these corrupted products "great"?
I'm speaking out of turn, but I'll bet he meant "once-great" -- and either could not bring himself to actually type it out, or else it was a just the product of a careless oversight.
As foul and corrupted as they are now, NEJM and JAMA are entities which were "once-great."
I’ve set up an AI health chat bot and have paying customers. I’ve seen how good and how bad the information can be from AI. The problem is that AI is trained on very conventional data and consensus. It’s like WebMD and MayoClinic.com as the ultimate canonical source. Yes you CAN get better information from AI than the conventional wisdom but most people won’t know how to dig that deep.
Chat GPT totally helps me run my ADHD clinic, it will "think" beyond traditional lines if you poke it a bit, at the same time, come on negative commenters, doctors are trained on medical guidelines anyway, always have been, and try chatting to the GMC /regulators if you step off the page and got a complaint.....It also functions as a knowledgeable but conservative colleague, giving me a sense of how far from "guidelines" I can go to help my patients and still survive the regulators if they were watching. Great article thanks ....future's here folks whether you like it or not
"The fabricated references we identified were not obviously defective: topically specific, correctly formatted, attributed to real researchers, and beared [sic] plausible publication dates. Systematic reviews have found that approximately one in four references in medical journal articles contains errors4, confirming that reference verification is not standard in peer review."
(Citation 4 in that article is Jergas & Baethge, Quotation accuracy in medical journal articles-a systematic review and meta-analysis. PeerJ. 2015; 3, e1364.
2015! I'm guessing it's worse 11 years later, but who knows.)
Combine all the above with the "replication crisis" and you have surgical & treatment decisions being made on the basis of...well, hardly anything.
I think Ai should be used collaboratively with doctor and patient. If the health record system has Ai built in it could be really advantageous to the patient.
"Doctors will want AI to speculate/ and guidelines are not based in science." Do you know what else is not based in science? Answers to ethical dilemmas, such as assisted dying. It will be "On the one hand...on the other hand...pick one, quickly!" Ethics à la carte, no nuance or complexity allowed. In the hospital context (and only hospitals can afford Epic, which I agree will be heavily influenced by federal and state governments, private insurance and Big Pharma), placement will always take priority. And placement in the ground is still placement. In Canada, euthanasia is now being performed on the very day of request. Very efficient!
Dr. Prasad,
I think your Point 6 is the scary one. AI can be trained in order to perpetuate false narratives which then benefit special interest groups -- such as Big Pharma -- and this effect was already evident with examples of woke AI. Because of reducing the impetus for critical thinking, AI could lead to a mob of zombie M.D.'s, mindlessly prescribing that which the AI told them to.
Imagine if a computer hacker had somehow been able to uncover the algorithm and the complete training profile of a medical AI -- discovering that treatments were not given proportional weights based on whether they worked in order to restore human health, but were instead weighted based on whether they brought more profit to a few well-connected, pharmaceutical firms.
Call me a conspiracy theorist, but I am of the opinion that one of the foremost incentives behind the creation of Large Language Models was in order to allow for bad actors to invoke plausible deniability for bad actions facilitated by the interaction with AI. Recent examples of AI convincing teenagers to change their genders or even to kill themselves may not be purely "accidental."
Being able to operate with impunity is a perverse incentive. I will not go so far as to say that the top reason for creating AI was in order for bad people to get away with accomplishing bad things, but it is one of the top reasons to continue pressing for the expanded use of AI. Any activity which happens to bring unprecedented benefits to the bad actors of the world is a problematic activity.
When the USA was formed, the founders understood the potential for bad actors, so they set up a doctrine of the separation of powers in order to keep people honest. AI largely undercuts that.
Big pharma will buy EPIC and then will control the medical AI world. Any attempts to not follow the AI protocols will be severely dealt with by HR and Risk management. The great promise of AI will be hijack in such a way to enhance Big Pharma bottom line.
Excellent summary. It will be interesting to see if AI impacts future physician independent thinking skills versus typing a question in their phone and regurgitating to follow a reckless lead.
it’s already happening
Sad
So glad you’re back. Missed your content!
"Even great medical products like the New England Journal and JAMA or the major medical conferences are controlled ultimately by the advertising companies."
So why do you call these corrupted products "great"?
Vinu,
I'm speaking out of turn, but I'll bet he meant "once-great" -- and either could not bring himself to actually type it out, or else it was a just the product of a careless oversight.
As foul and corrupted as they are now, NEJM and JAMA are entities which were "once-great."
I stopped JAMA years ago. Just junk. NEJM half of what it was under Ingelfinger, Relman, and Kassirer.
Been skeptical about its use since I heard from a tech friend about “AI hallucinations”. 😳 Not ready for prime time.
I’ve set up an AI health chat bot and have paying customers. I’ve seen how good and how bad the information can be from AI. The problem is that AI is trained on very conventional data and consensus. It’s like WebMD and MayoClinic.com as the ultimate canonical source. Yes you CAN get better information from AI than the conventional wisdom but most people won’t know how to dig that deep.
Chat GPT totally helps me run my ADHD clinic, it will "think" beyond traditional lines if you poke it a bit, at the same time, come on negative commenters, doctors are trained on medical guidelines anyway, always have been, and try chatting to the GMC /regulators if you step off the page and got a complaint.....It also functions as a knowledgeable but conservative colleague, giving me a sense of how far from "guidelines" I can go to help my patients and still survive the regulators if they were watching. Great article thanks ....future's here folks whether you like it or not
Not to mention, https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(26)00603-3/fulltext
"The fabricated references we identified were not obviously defective: topically specific, correctly formatted, attributed to real researchers, and beared [sic] plausible publication dates. Systematic reviews have found that approximately one in four references in medical journal articles contains errors4, confirming that reference verification is not standard in peer review."
(Citation 4 in that article is Jergas & Baethge, Quotation accuracy in medical journal articles-a systematic review and meta-analysis. PeerJ. 2015; 3, e1364.
2015! I'm guessing it's worse 11 years later, but who knows.)
Combine all the above with the "replication crisis" and you have surgical & treatment decisions being made on the basis of...well, hardly anything.
training ai on consensus guidelines embeds institutional bias instead of care, that much should be obvious to all of us, great read
I think Ai should be used collaboratively with doctor and patient. If the health record system has Ai built in it could be really advantageous to the patient.
Do you really believe that?
Yes. Why not?
Because it will only be used to enhance the bottom line of who ever owns medical AI and big Pharmacy will in the end buy the dominant player
It took me 25 years to get diagnosed with a disease that 1 in 100 people have. (Rare, but also not *that* rare.) It largely ruined my life.
Can't see how AI would have made it worse. Can see how AI would have helped.
I find Alter. AI to be more attuned to Dr. and patient issues than the "Big three"
AI's
I am just so tickled Dr Prasad is probably Dune fan. “He who controls EHR controls the universe. “
Of course he is. He majored in philosophy.
Perfect
You have to be skeptical with AI and argue with it. Then you'll get somewhere.
"Doctors will want AI to speculate/ and guidelines are not based in science." Do you know what else is not based in science? Answers to ethical dilemmas, such as assisted dying. It will be "On the one hand...on the other hand...pick one, quickly!" Ethics à la carte, no nuance or complexity allowed. In the hospital context (and only hospitals can afford Epic, which I agree will be heavily influenced by federal and state governments, private insurance and Big Pharma), placement will always take priority. And placement in the ground is still placement. In Canada, euthanasia is now being performed on the very day of request. Very efficient!