Study: Hospital rankings may rely on faulty data

The October 2016 issue of The Joint Commission Journal on Quality and Patient Safety, features the article "Consumer Rankings and Health Care: Toward Validation and Transparency" by Bala Hota, MD, MPH, and co-authors, Rush University Medical Center (RUMC), Chicago. After RUMC received a lower than expected ranking for patient safety in the 2015-16 U.S. News & World Report's (USNWR) "Best Hospitals" rankings, the authors compared the data that USNWR used for their hospital to their own internal data.

The authors found that the USNWR data showed many more patient safety events, such as pressure ulcers, almost all of which had actually been present at the patient's admission. Suspecting a broader problem, Hota et al. analyzed data on a sample of hospitals and found that RUMC was not the only organization with discrepancies in data. False-positive event rates were common among high-transfer and high-volume hospitals.

The authors conclude more transparency and validation is needed for consumer-based benchmarking methods. In response to these findings and concerns raised by others, USNWR made changes to its methodology and data sources in 2016.

In an accompanying editorial, "The Quality Measurement Crisis: An Urgent Need for Methodological Standards and Transparency," David M. Shahian, MD, Elizabeth Mort, MD, MPH, and Peter J. Pronovost, MD, PhD, reflect on the Hota et al. article to conclude, "Just as health care providers have ethical and moral responsibilities to the public they serve, rating organizations and journalists that grade providers have similar obligations--in their case, to ensure measure validity and methodological transparency."

RUMC further explored the importance of rating organization's validity and methodological transparency in the following essay, "Hospital Rankings Have Room for Improvement

Hospitals that care for sickest of sick found to be unfairly ranked

When consumers make what can be life-or-death decisions about where to seek health care, they often look to the numerous online hospital rating services that are available, including the government-sponsored Hospital Compare, Consumer Reports, and Leapfrog.

Rating hospital quality is not as simple as rating the quality of a car, though. One reason is that patients vary in terms of how sick they are when they arrive at the hospital or undergo surgery. Older patients with multiple health problems or advanced diseases are more likely to develop an infection or complication than younger, healthier patients -- even though they receive the same quality of care.

The best hospital rating systems try to account for severity of illness. However, a Rush data expert's recent analysis of U.S. News & World Report's "Best Hospitals" rankings shows that these rating systems often are dependent on faulty data sets and performance metrics.

Rush University Medical Center consistently has scored high on patient safety measures, according to most rankings. In fact, Rush University Medical Center has received nine consecutive "A" grades for safety from the Leapfrog Group, putting it among fewer than 100 hospitals in the country that have received the highest possible grade each time since Leapfrog began issuing the semi-annual ratings in 2012.

But last year U.S. News "Best Hospitals" issue gave Rush the publication's lowest ranking for patient safety (one out of five). To learn what was behind the discrepancy, Rush's quality team - Tom Webb, Bala Hota, MD, MPH, and Omar Lateef, DO - dug into the U.S. News data and methodology.

"When we compared the data that U.S. News used for our hospital to our own internal data, we found big differences," says Hota, Rush's chief research informatics officer. "U.S. News showed many more patient safety events than our actual rates."

For example, U.S. News data showed that 25 patients at Rush had developed pressure ulcers, or bed sores, over a particular time period. In fact, only one patient had developed a pressure ulcer while at Rush.

Suspecting a broader problem, they analyzed data on a sample of hospitals and found that Rush was not the only organization inaccurately unfairly ranked. Hota reported the results of his study in the October issue of the Joint Commission Journal on Quality and Patient Safety.

Two hospital characteristics tended to negatively skew U.S. News patient safety scores: Taking care of large numbers of patients (i.e., high volumes) or admitting a high percentage of patients transferred from other hospitals.

Webb, Hota and Lateef found that the U.S. News data set was missing key information, such as whether patients with pressure ulcers had these sores when they were admitted to Rush. If conditions were present prior to arrival, Rush should not have been penalized.

Large referral hospitals such as Rush, which provide highly specialized care for the sickest patients and most complex conditions, also tend to have high volumes and high transfer rates. "Rush is the biggest transfer center in Illinois," Hota says. "If somebody is sick in a community hospital and not getting better, that hospital knows Rush is a reliable place to transfer the patient to."

Rush notified U.S. News about the Rush findings in 2015and U.S. News made some revisions to its data methodology. As a result, Rush's patient safety score increased to four out of a possible five in the 2016-2017 U.S. News ratings released in August.

"U.S. News was very responsive to our findings, and the Rush community should take pride in the fact that we take care of very sick patients very well," says Lateef, Rush's chief medical officer. "When you adjust for patient acuity (severity of patients' conditions), our outcomes data are among the best in the country."

The Rush analysis of the U.S. News ratings confirms what many experts have suspected. "People have argued and theorized that if you take care of a lot of very sick people or have a lot of transfers, then the data could be biased," Hota says. "This study proves that. It shows that data is not perfect, and misunderstandings in the data can negatively portray a hospital."

Acknowledging that hospital rankings have room for improvement, U.S. News is convening a panel of experts in November to discuss how to better evaluate hospital quality, safety and performance, according to a press release.

Lateef and Hota, and the quality team believe that right now the rules used by hospital rating systems are too complex for most consumers to navigate. That's why they are urging U.S. News and other hospital rating systems to be more transparent about how they rate hospital quality, so that individual hospitals can validate the accuracy of their scores and work with the rating organization to adjust the scores as appropriate.

"The companies that are producing these ratings have to be more transparent," Hota says. "Their methodology should be reproducible and easily understood."

In the meantime, Rush will continue to measure, track, and share patient safety data rigorously, and use that date to drive improvements to make Rush's care even safer. "We consider quality to be the most important part of patient care," Lateef says. "We believe in being transparent about quality so we can continue marching towards improvement."

###