Key measure of hospital quality does not give accurate indication of avoidable deaths

Standardised mortality ratios (SMRs) for hospitals do not provide an accurate picture of how many deaths could have been avoided, according to a new study by researchers at the London School of Hygiene & Tropical Medicine and Imperial College London published in the BMJ.

The authors say the widely used hospital-wide standardised mortality ratios, such as HSMR (Hospital Standardised Mortality Ratio) and SHMI (Summary Hospital level Mortality Indicator), should not be used to benchmark hospitals' quality of care.

Hospital-wide SMRs compare the number of deaths in a hospital with the expected number (based on a statistical probability derived from routine administrative data). These measures have been widely used in many countries for more than 20 years and are often used to identify 'problem' hospitals.

The study was commissioned by Professor Sir Bruce Keogh (Medical Director of NHS England) following his review of 14 Trusts in 2013 which questioned whether SMRs for hospitals provide an accurate indication of the number of avoidable deaths occurring. It was funded by the National Institute for Health Research and the Department of Health.

Led by Dr Helen Hogan and Professor Nick Black (London School of Hygiene & Tropical Medicine) and Professor the Lord Ara Darzi (Imperial), the study is the largest nationally representative case record review of hospital deaths in England, based on 3,400 deaths in 34 hospital trusts randomly selected from across the SMR spectrum. [1] Experienced doctors reviewed patients' case notes and the proportion of deaths they judged likely to have been avoidable [2] was compared with the trust's HSMR and SHMI.

The researchers found that the proportion of avoidable hospital deaths was 3.6%. There was no significant association between hospital-wide SMRs and the proportion of avoidable deaths in a trust. [3]

Lead author Dr Helen Hogan, Senior Lecturer in Public Health at the London School of Hygiene & Tropical Medicine, said: "Our findings suggest that the commonly used hospital-wide SMRs are not a useful reflection of the proportion of avoidable deaths in a trust.

"Dreadful though each avoidable hospital death is, they are too infrequent to be the basis of a robust indicator to detect significant differences between trusts. There are credible alternatives for assessing the quality of hospitals which give a fairer and more accurate picture."

Study senior author, Nick Black, Professor of Health Services Research at the London School of Hygiene & Tropical Medicine, said: "Hospital-wide SMRs can be distracting and potentially misleading and should not be used as a basis to praise or condemn a hospital or trust. We need to support and help hospitals to raise standards not criticise and punish them.

"Given the complexity of hospitals, with many different clinical departments and activities, it is more helpful for the public, patients, staff and politicians to use a variety of specific measures of quality, such as adherence to good practice guidelines; outcomes for specific diseases or procedures; patient surveys of their experiences; infection rates; and staff surveys."

Fellow study author, Ara Darzi, Director of Imperial College London's Patient Safety and Translational Research Centre and Chair of Imperial's Institute of Global Health Innovation, said: ''Quality must remain the organising principle of our National Healthcare System. We know that the measurement of patient safety within a hospital or clinical environment can be a useful benchmark for assessing quality in health care and our study highlights the importance of clinician-led case-note reviews as a robust and invaluable methodology for determining such quality."

The authors recommend that staff continue to review individual deaths to identify local problems with care but reviewing should be standardised so that all acute hospital trusts adopt a rigorous approach and take action to prevent future deaths. They suggest SMRs should only be used to assess the quality of care for those conditions with high case fatality for which good quality clinical data exist (such as critical care patients).

The researchers note some limitations to the study, including how the lack of a significant association between hospital-wide SMRs and avoidable death proportions may partly reflect the methodological limitations of both type of measure.

###

For more information or to request interviews contact the London School of Hygiene & Tropical Medicine press office on press@lshtm.ac.uk or +44(0)2079272802.

Helen Hogan, Rebecca Zipfel, Jenny Neuburger, Andrew Hutchings, Ara Darzi, Nick Black, Avoidability of hospital deaths and association with hospital-wide mortality ratios: retrospective case record review and regression analysis. BMJ. DOI: 10.1136/bmj.h3239

Once published the paper will be available at: http://www.bmj.com/cgi/doi/10.1136/bmj.h3239

A linked editorial will be published alongside the paper and will be available at: http://www.bmj.com/cgi/doi/10.1136/bmj.h3466)

[1] Data for 10 trusts in 2009 and 24 trusts in 2012/13.

[2] Avoidable deaths were those judged by trained medical reviewers to have at least a 50% likelihood of having been avoidable through acts of omission or acts of commission.

[3] Association of avoidable death proportion with Hospital Standardised Mortality Ratio (HSMR) and Summary Hospital-level Mortality Indicator (SHMI) assessed using regression coefficients which estimate the increase in avoidable death proportion for a one standard deviation increase in SMR.

Researchers found that a difference in HSMR of 105 and 115 might be associated with an increase of only 0.3% in the proportion of avoidable deaths (although this estimate was not statistically significant). A similar non-significant association was observed for the other hospital-wide SMR used in England, the Summary Hospital-level Mortality Indicators (SHMIs).

The study was funded by the National Institute for Health Research (NIHR) Research for Patient Benefit (RfPB) Programme (PB-PG-1207-15215) and the Department of Health Policy Research Programme (DH PRP).

The London School of Hygiene & Tropical Medicine is a world-leading centre for research and postgraduate education in public and global health, with 3,900 students and more than 1,000 staff working in over 100 countries. The School is one of the highest-rated research institutions in the UK, and among the world's leading schools in public and global health. Our mission is to improve health and health equity in the UK and worldwide; working in partnership to achieve excellence in public and global health research, education and translation of knowledge into policy and practice. http://www.lshtm.ac.uk

The National Institute for Health Research (NIHR) is funded by the Department of Health to improve the health and wealth of the nation through research. Since its establishment in April 2006, the NIHR has transformed research in the NHS. It has increased the volume of applied health research for the benefit of patients and the public, driven faster translation of basic science discoveries into tangible benefits for patients and the economy, and developed and supported the people who conduct and contribute to applied health research. The NIHR plays a key role in the Government's strategy for economic growth, attracting investment by the life-sciences industries through its world-class infrastructure for health research. Together, the NIHR people, programmes, centres of excellence and systems represent the most integrated health research system in the world. For further information, visit the NIHR website (http://www.nihr.ac.uk).

The Department of Health (DH) helps people to live better for longer. The Department leads, shapes and funds health and care in England, making sure people have the support, care and treatment they need, with the compassion, respect and dignity they deserve. The Department funds health research and encourages the use of new technologies because it's important to the development of new, more effective treatments for NHS patients. Innovation is needed so that decisions about health and care are based on the best and latest evidence. http://www.dh.gov.uk

The Department of Health's Policy Research Programme (PRP) commissions, funds and manages leading edge research that is focused on the needs of Ministers and policymakers for evidence to inform policy. PRP provides timely evidence for current policy needs and builds the evidence-base for future policy-making. http://www.prp-ccf.org.uk/