<img height="1" width="1" style="display:none;" alt="" src="https://dc.ads.linkedin.com/collect/?pid=152233&amp;fmt=gif">
❮ Back to Blog

Want to Benchmark Your Hospital or Medical Practice’s Performance Metrics? Here is How You Identify Appropriate Peers for Comparison

by Intermedix Staff on December 19, 2017 at 12:44 PM

blog-benmarking-against-peers-identifying.png

When reviewing practice performance metrics, leaders frequently ask how they stack up against their peers. Answering this question requires addressing two separate analytic problems. First, how can we determine who our peers are? Second, can we obtain data on the outcomes of interest from our peers once we identify them?

This post covers how we can use publicly available data from the Centers for Medicare and Medicaid Services to answer the first question. Then, we will use this data to find the group of hospitals most similar to a randomly chosen large, nonprofit, teaching hospital (Hospital X) and compare how our conclusions on successful sepsis treatment change when comparing Hospital X against its peers rather than other hospitals in its same state.

Clustering using CMS Cost Report Data

In a previous post Making the Most of Free Data for Healthcare Analytics, I highlighted the availability and utility of the CMS Cost Reports. As a brief recap, CMS publishes self-reported financial, staffing and patient volume data on every hospital that treats Medicare patients. Hospitals report on the non-Medicare population as well. Using this data, I pulled bed counts, patient days and discharges for overall, Medicaid and Medicare populations at each hospital. I also extracted information on annual revenue and expenditures on facilities and practitioners, as well as utilization of a hospital’s ICU, burn unit and maternity ward. Finally, I pulled information on whether a hospital is a for-profit, nonprofit or government-run facility and is a teaching hospital. CMS reports this information on over 6000 hospitals, but I used a subset of roughly 3300 acute care facilities. This sample excluded rehabilitation hospitals, long-term care facilities and psychiatric hospitals.

After obtaining this data, it was divided into ‘most-similar’ segments. There are several ways to do this, but a basic K-Means cluster was used. To make a direct comparison to state-based group averages, I segmented the data into 50 clusters. A machine learning algorithm then iterated through all the hospitals and assigned them into the clusters such that intra-cluster variation was minimized and between-cluster differences were maximized.

Hospital X was assigned to a cluster with five other hospitals so as to see how these hospitals compared against all other Tennessee hospitals.

 

Hospital X

Other Hospitals in X’s State

Cohort

# Hospitals in Group

1

91

5

# Teaching Hospitals in Group

1

15

5

# Nonprofit Hospitals in Group

1

37

0

# Government-Run Hospitals in Reference Group

0

15

5

Average Bed Count

567

142.99

355.4

Average Medicaid Discharge as % of all Discharges

24.72%

14.14%

19.54%

Average Length of Stay

3.30

3.71

3.66

The algorithm determined that Hospital X belongs in a relatively small cluster with five other teaching hospitals. These all are affiliated with state schools and include the University of California – San Diego Medical Center and the University of Iowa Hospitals and Clinics. These hospitals tend to have high bed counts closer to Hospital X’s size and have higher Medicaid treatment rates. A patient’s average length of stay at these facilities is also closer to Hospital X’s than the state average.

Benchmarking Sepsis Death Rates

I obtained sepsis mortality rates for Hospital X, its benchmark group and all the other hospitals in its state from www.healthgrades.com. HealthGrades calculates mortality rates based on Medicare claims files, so the data is consistently measured and available for all US hospitals. Our metric was the 30-day mortality rate. That is, among patients who have a diagnosis of sepsis at some point during an inpatient stay, what proportion died during their stay or within 30 days of discharge? Hospital X’s 30-day mortality rate was 28.05%. While ideally this number would be 0, it is useful to know whether this number is relatively high or low.

Using averages for the state, we can make an initial comparison. The statewide average 30-day mortality rate is 21.93%. The difference in these figures suggests that Hospital X lags behind its neighbors in its treatment of sepsis. However, there may be interdependence in how successful these other hospitals are hospitals are – local hospitals might refer their hardest cases to Hospital X, or its reputation might make sick patients more likely to seek care there by default. When the sickest patients are admitted to Hospital X, they will not count against other nearby hospitals’ performance metrics.

We can instead compare Hospital X’s mortality rate against the average of its peers. The average sepsis mortality rate among its five best matches is 17.50%. This suggests Hospital X is doing worse in comparison to the average. When compared to the nationwide hospitals most similar to it, Hospital X does very poorly in treating sepsis.

Conclusion

These results only tell part of the story. There are unaccounted-for causes of sepsis and mortality that contribute to these death rates. While we can factor in patient insurance information, the CMS aggregate data makes it difficult to directly risk-adjust for patients’ health status. In addition, the clustering algorithm picked Hospital X’s best matches by treating every factor equally – a business user could instead adjust the model to emphasize patient population characteristics, hospital size and finances or specialty services. Any changes to these weights would likely change the pool of hospitals the model determines are most appropriate for benchmarking. By performing the comparison several times under different specifications, though, we are more convinced that Hospital X has room for improvement in treating patients with sepsis. This allows for a more detailed comparison than simply comparing a hospital against its regional or national averages.

New Call-to-action

Recent Posts

author avatar

This post was written by Intermedix Staff