How U.S. News puts together its ‘Best Hospitals’ rankings

Each year, U.S. News & World Report publishes its much-anticipated ranking of America’s top hospitals, but the methodology changes each year.

U.S. News unveiled its 2023-24 “Best Hospitals” Tuesday morning, and this year’s analysis includes a number of changes, including more ratings on outpatient care, and expanded consideration of health equity, the COVID-19 pandemic and others. (See the full U.S. News rankings.)

Ben Harder, the U.S. News chief of health analysis, says that while the revisions in methodology may not represent an all-time record, “I think we've made more changes this year than we tend to.”

In a recent interview with Chief Healthcare Executive®, Harder discusses the work involved in assembling the rankings, including the changes this year. And he responds to some criticism directed at the annual report.

“We're looking to keep pace with how care is being delivered,” Harder says.

(Ben Harder of U.S. News talks about the rankings in this video. The story continues below.)

This year, U.S. News has added outpatient outcomes in orthopedic and urology, and it’s possible that other outpatient measures will be added in the future.

U.S. News is also giving greater attention to health equity, as this is the third year that hospitals are evaluated in equity. This year, U.S. News is incorporating more data about the members of underserved minorities in their patient population in order to more accurately gauge the performance of hospitals. And the rankings include an “Area Deprivation Index” to weigh socio-economic challenges along with outcomes.

“Some of those changes to the equity measures this year came out of conversations that we had with equity experts and other you know, other clinical experts,” Harder says.

U.S. News says it is placing more weight on objective quality measures and is reducing its reliance on expert opinion.

U.S. News ranks hospitals in 15 specialties, and utilizes objective data (such as data from Medicare) in 12 specialties, including cancer, heart surgery, and obstetrics and gynecology. U.S. News relies on expert opinion for three specialties: ophthalmology, psychiatry and rheumatology.

“I think we have always valued outcomes,” Harder says. “Patients value them, clinicians value them, because it's something that they have the opportunity to improve on, and it's meaningful for their patients,” Harder says.

“Fifteen years ago, I would say, expert opinion was a pretty big determinant in our results,” he adds. “I don't think that's true anymore.”

Eventually, Harder says he expects U.S. News will stop using expert opinion in its analysis of the top hospitals, but it may take some time.

“I would say at some point, it will be dropped,” he says. “The reason you might not want to hold your breath for it is that there are some domains of care where really objective measures of quality are lacking or are not well validated.”

U.S. News has received some criticism over its rankings of hospitals.

Penn Medicine said in June it would “discontinue active participation” in the annual rankings. Penn Medicine said the rankings focused too much on inpatient care, even as more care is done on an outpatient basis. Penn also faulted the rankings for its reliance on Medicare data, meaning the data covers patients who are 65 and older or have long-term disabilities.

St. Luke’s University Health Network in northeastern Pennsylvania said in late May that it would step away from the rankings, pointing to the use of subjective opinion in the analysis. Both Penn and St. Luke’s said they would stop marketing their placement in the U.S. News rankings.

Harder says it’s important to recognize that U.S. News isn’t soliciting hospitals for information in determining the rankings. U.S. News is using publicly available data and, to a degree, expert opinion in assessing the hospitals.

“We don't survey hospitals for the rankings,” he says. “It's all third party data … they're not opting in, they're not opting out.”

U.S. News evaluates any hospitals that are receiving any taxpayer dollars, he notes.

“For us, it's really important that we're able to publish information for patients for all hospitals,” Harder says. “And for that reason, we don't depend on hospitals’ submitted data.”

“We have always had a very open dialogue with hospital leaders, about their perceptions of the methodology, and where they have concerns or critiques or constructive feedback for us, we've been very receptive to that,” Harder says.

“Any feedback, constructive or otherwise, in any venue, we're going to absorb, we're going to think about, we're going to take into account and see what we can do to address it, as long as it improves the utility of our product for patients, because that's our core audience,” he adds.

Harder says one of the most complex factors in the rankings has been accounting for the COVID-19 pandemic. Some hospitals dealt with patients with more cases, and some had patients with more serious illness. The rankings exclude patients admitted with COVID-19 in 2020. If a patient was diagnosed in 2021, the data is included but risk-adjusted.

While Harder says the methodology changes every year, he says the focus remains on patients. Health system executives have said patients are taking more time in studying doctors and hospitals, just as consumers devour information before buying a car or a television.

Some readers are doing quick reviews of hospitals, but others take deeper dives, Harder says. Some of the most active and knowledgeable users are parents of sick children who are assiduously reviewing the U.S. News database to gather as much information as possible.

“They know their data backward and forward,” Harder says.

Millions of consumers are using the “Best Hospital” rankings, and that’s an indication that patients are increasingly engaged and want more information before deciding on where they will get their care.

“I think it's a victory for data transparency that patients are increasingly aware of the availability of quality information, and increasingly discriminating about using that as one of the ingredients in their decision making,” Harder says.