Healthcare AI Must Use Real-World Data to Avoid Generalization

“Any doctor who could be replaced by a computer probably should be,” Martin Kohn, MD, joked. The technology isn’t there yet, for several reasons.

Kohn used to be the chief medical scientist for IBM Research, and he worked on Watson for some time. Now an independent clinical informatics consultant, he said most of the current capabilities of artificial intelligence (AI) in healthcare are focused on clinical decision support, and they are limited. Had Watson or any other project had a profound effect yet, it would be published in a peer-reviewed scientific journal somewhere.

Not that he puts too much stock in those. One of the limiting factors for Watson and other AI technologies is that their recommendations are based on all of the journal articles and guidelines they were fed from the start. As other speakers at the AI for Healthcare Summit in Boston, Massachusetts pointed out today, half of published research findings are inapplicable or false.

Journals used to be the gold standard for medicine, Kohn said, but now that innovations come faster and faster and healthcare has the capability—and the obligation—to personalize care, articles and studies are becoming less relevant. Studies almost always focus on patients with a similar disease, but given the uniqueness (like presence of comorbidities) in each patient, that may not be the most informative route.

Consensus guidelines, he said, are inhibitors to innovation: Less than a fifth are truly evidence-based. Kohn called the very concept of standard-of-care “an obstacle to personalized healthcare.”

Medicine has to adapt and use the wide array of data it can gather about a single patient to treat that patient, rather than asking a machine to suggest ways that have worked for others. Medical school training has to challenge students to understand this shift.

“It’s a fundamental change in the concept of evidence,” he said. “We have to use AI on real-world data.” When he began his medical education in the 1960s, the challenge was getting data. Now, that’s no issue: The problem is analyzing all of it. That, he believes, is where AI will provide the most benefit.