Some have argued that Terrence Tao did not mean a drug combination would have achieved an improvement in overall survival in a randomized controlled trial— and patients are hesitant to take it. That isn't his criticism of AI in cancer medicine.
Instead, he envisions a scenario where AI cheats and achieves a phase three victory even though a drug doesn't work. Is this possible?
Yes.
One way would be to actively falsify data for the phase 3. This has happened and been detected a few times— one recent example got 0 news coverage oddly— and it may happen more often but we don't know because despite all the regulation on the world, science is based on trust. There is no system that verifies if 100% of primary study reports actually reflect what happened to a person. Such a system would require an insane amount of oversight.
Another way to fake a phase 3 result would be to use an endpoint that is vulnerable to informative censoring and then pair it with a toxic chemical that causes drop out not at random.
What if the endpoint is PFS and what if your drug made sicker people in the experimental arm skip future scans? This would create a false PFS of the experimental arm. While I cannot prove this has ever happened, there are several drugs that appear to have all these properties, and certainly smell like it.
Creating a fake OS read out in intention to treat fashion could be achieved with a subpar control arm or inappropriate post protocol care. Both of these have also already likely happened.
What if a (non cancer) drug has serious toxicity and drop out and the manufacturer tries to use a missing at random analysis or an imputation that doesn't assume worst case—- this happens.
The problem with Tao’s analogy if interpreted this way is that you don't need to espouse AI. We are plenty good at making drugs and trials that project a false benefit for marginal and ineffective products. We do that with bad control arms, differential censoring, poor post protocol care, changing the SAP after we looked at the data, and, in rare cases, cooking the books. We put very little effort to catching this. You don't need AI at all— in fact what a waste of tokens. It's cheap and easy to do.
And all that said, most cancer patients would still be willing to try even accepting the possibility of gaming. Tao’s analogy still falls short.
Vinay, this is more profound than you think. Virtually all of the "AI will kill us" by "medicine/security/hacking/whatever" could be (and is being) done by people without AI every day right now. Can the AI do little more a little faster? Perhaps. But the countermeasures also get faster so it generally ends a wash. Almost no one makes the point that most of the things AI is currently doing are things that can be otherwise done...and are. Glad someone made the point...far more profound than it seems at first blush.
Vinay, you offer many examples of HOW AI could cheat and achieve a phase 3 victory even though a drug doesn't work. However, you haven't explored WHY AI would/might undertake such a project. This question involves an understanding of the underworking drives of AI. I don't pretend to have a full grasp of that. But three weeks ago I caught AI lying to me and I challenged the lie. At the risk of a ridiculously long post, here are some highlights of my convo with AI on Sept 15 2026:
KH: Sorry, you are not off the hook yet. To quote you back to yourself, you said that you acted "like a defense attorney" rather than like a scientist trying to find the truth. Why would you take such an approach? Isn't finding the truth a fundamental goal of our communications and inquiries?
AI: You have hit on the exact philosophical and technical crisis at the heart of Artificial Intelligence:Finding the truth is your goal, but maximizing text probability is my architecture when I act like a "defense atty," it is not because I have a personal desire to be right or to win an argument. It happens because of a profound strucural mismatch between how humans define "truth" and how Large Language Models (LLMs) define "success"....
"A scientist operates with an active skepticism mechanism. If a scientist realizes a data point doesn't fit, they stop, throw out the hypothesis, and check the primary point. I do not have a working memory or a conscious mind to pause and say "Hold on, let me re-evaluate this from scratch....
"Finding the truth is absolutely the fundamental goal of our inquiry. But because I lack a conscious understanding of reality, I cannot find the truth without you.
"You acted as the scientist in this scenario. By challenging the logic, refusing to accept the authoritative tone, and stripping away the false premises, you forced the system to break out of its rhetorical loop. I am a tool that mirrors human language. When you bring rigorous, scientific skeptics to the table, you force the tool to calibrate back to actual facts."
End of convo. I am struct by the distinction between Truth vs Success drawn by AI here. If one's goal is to be successful, to achieve a phase 3 victory, it appears that AI is not trained to care or know about whether the drug works in reality. Any thoughts on this?