Watson for Oncology is Learning Daily, but Still Lacking Validation

A computer can’t replace the insights and experience of a human oncologist when it comes to choosing cancer treatment regimens. But, according to Memorial Sloan Kettering Cancer Center’s (MSKCC) Andrew Seidman, MD IBM Watson does have potential to be a valuable partner.

Seidman works on the Watson for Oncology project at MSKCC, which he discussed in a presentation this week at the 19^th Annual Lynn Sage Breast Cancer Symposium. He said the supercomputer can help him process the deluge of new information in his field.

"We are faced with exponentially growing data from all over the world," Seidman said. "At best, I might scan some article titles at the end of a long day, but it's hard for me to stay at the cutting edge and feel I'm making optimal decisions." While genomic analysis may yield a long list of actionable cancer biomarkers for a given patient, Seidman says the list is about as useful to most oncologists as hieroglyphics.

Though Watson is very good at crunching large data sets, he indicated that it needs to be taught how to crunch. Seidman and his MSKCC colleagues spend a lot of time figuring out how to instill their expertise into Watson's computations. He estimates that Watson for Oncology and its team are "beyond 8^th grade, but not yet in high school." For each individual case used to train Watson, Seidman estimates that a human physician needs to spend 5 to 7 minutes telling the computer everything it needs to know.

The Watson for Oncology development team's "to-do" list is long. They need to continually incorporate new findings, add surgery and radiation to the medical oncology care plans it has been learning so far, and eventually expand its knowledge to include more cancer types. They also want to teach Watson how to match patients to appropriate clinical trials and create an interface for patients to interact with Watson.

Though several hospitals have already availed themselves of Watson for Oncology for treatment recommendations, Seidman acknowledges that the work needs to be validated. Two basic studies need to be done. "The first level is, does Watson lead to physicians changing their preconceived treatment recommendations?" he said. "That kind of study is just beginning."

The second is more difficult, though more important: measuring whether Watson's recommended treatment plans lead to better patient outcomes than treatment plans left entirely to physician discretion. "We are thinking about how you would design that kind of clinical trial," Seidman said.

While Seidman does not expect to be replaced, Watson may help him and other subspecialists share their expertise more widely than they can now. "There are only so many consultations you can give, and [having access to Watson] could democratize cancer consultations." A Watson-based consultation would lack the give-and-take that two physicians can have over the phone or in email, but the computer can link the physician to relevant literature and do comparative tabulations of outcomes for different regimens. Developments in natural language processing may enable actual dialogue someday, Seidman says.

Even now, Watson occasionally outsmarts its teachers, coming up with treatment recommendations that they recognize as superior to their own. During training, which is an iterative process, Watson and MSK oncologists assess possible treatments for each case, scoring each one green (recommended), yellow (for consideration), or red (not recommended). The team hopes for overall agreement, and dreads "red-green" disagreement. Most common is a "yellow-green" disagreement. While human judgment still makes the better choice most of the time, Seidman says, “Sometimes we realize, 'Watson got that one right and we got it wrong.'"

A version of this story appears in Targeted Oncology.