How Phoenix Children’s Hospital has succeeded with machine learning | HIMSS 2023

Chicago - David Higginson says many experiments with machine learning models aren’t going to be successful.

David Higginson, chief innovation officer of Phoenix Children's Hospital, talks about the use of machine learning during a session at the HIMSS Conference. (Photo: Ron Southwick)

In fact, from his experience, he said, “Three out of four models completely fail.”

But Higginson, the executive vice president and chief innovation officer at Phoenix Children’s Hospital, says the projects that succeed can improve patient care and yield financial returns.

He shared some of the successes Phoenix Children’s has seen with machine learning during a session at the HIMSS Global Health Conference & Exhibition Tuesday afternoon.

Using machine learning, Phoenix Children’s managed to identify kids that were suffering from malnutrition. In a typical week, six kids are now being diagnosed with malnutrition that may otherwise have been missed.

“If you're a physician in the ED, or a surgeon, and a patient comes to you, malnutrition isn't often the first thing you're thinking about,” Higginson said. “You're thinking about the broken leg, the symptom that's presenting to you.”

“We were convinced in this whole thing that people were getting missed, because the physician particularly wasn't thinking about it, especially if they were very busy, or maybe the disease they were being treated for was nothing to do with malnutrition,” he added.

So Higginson and his team developed an algorithm, using 25 years of data on patients with malnutrition. Then, the algorithm was put into the records of each patient that comes into the hospital, and the model predicted if the patient could be at risk of malnutrition.

The model was tested in the hospital, and dieticians were asked to see certain patients, without being told why. Eventually, the dieticians discovered a number of patients were suffering from malnutrition. Now, an algorithm is placed in the electronic health records and automatically signals patients at risk. (David Higginson shares more detail on machine learning at Phoenix Children's in this video. The story continues below.)

• Read more: AI in healthcare: 'Awe-inspiring benefits' and 'scary risks'

Begin with the end

The Phoenix Children’s emergency department also uses a machine learning algorithm to predict demand during the day. It’ll take into factors such as whether school is in session, and then staffing in the emergency department will be adjusted based on anticipated volume.

Beyond patient care, Phoenix Children’s has utilized AI to identify patients who may be more likely to skip appointments, and they book additional patients for those slots, allowing the health system to see more patients. The “no show” rate in the past has reached 20%, and Higginson says shaving one percentage point off that rate can mean $1 million in revenue.

Machine learning models have also been used to successfully identify people who would be willing to donate to the Phoenix Children’s foundation, he says.

Higginson offered several lessons in for hospitals that want to utilize machine learning.

For starters, Higginson told the audience, “Begin with the end in mind.”

Higginson suggested looking at problems to determine what can be predicted, and if an accurate prediction would make any difference.

“This is probably the hardest part in four or five years of trying to frame the problem as a prediction problem,” he said. “And you just have to be ready to say, ‘You know what, 80% of problems don’t fall in this bucket.’ Because you can't throw AI at everything.”

Once someone has an idea, ask what they would do if the model predicted 100% accurately, and how they would do it, he said.

Machine learning models aren’t going to reach 100% accuracy, or 90%, he said. For some problems in workflow management, a high degree of accuracy isn’t necessary, Higginson said.

“You’ll be surprised how valuable 50% accuracy can be in some cases,” he said.

Put another way, he said machine learning models don’t have to be perfect, but they can have value if they are at least as right as often as a physician.

• Read more: AI in healthcare requires 'approach of exploration'

Engage the operators

Health systems have to realize that machine learning models have a shelf life and can only last so long. “These models by themselves drift and don’t work anymore,” he said.

Higginson stresses that it’s critical to demonstrate the value of machine learning projects by measuring outcomes. As he said, “Not every cool idea can be implemented.”

It’s also worth adding that Higginson has no full-time employees. He is able to lean on other staff to help him with projects. But the success at Phoenix shows successful AI and machine learning projects don’t require an enormous staff.

Speaking after the session, Higginson also offered advice for health systems in developing machine learning models.

“Ask the operators, the people who will use it every day: Tell me how you'd use this,” Higginson said. “And work with them on their workflow and how it's going to change the way they do their job.

“And when they can see it and say, ‘Okay, I'm excited about that, I can see how it's going to make a difference,’ then go and build the model,” he added. “You'll have more motivation to do it, you'll understand what the goal is. But when you finally do get it, you'll know it's going to be used.”