Hunt’s proposed ranking of hospitals on avoidable mortality rates is a bad idea

Over the weekend it was announced that Jeremy Hunt wanted the NHS to tackle “avoidable deaths” in English hospitals (see this BBC report). On the face of it this seems like a good thing. Plans to review case-notes to see if anything could be learned, and then using these to establish a national rate of avoidable deaths appear to be perfectly sensible. However, the same cannot be said of the other part of the plan which is to rank hospitals according to avoidable mortality rates. There are a number of reasons why this might be a bad idea; how well can we tell if a death really was avoidable on a case by case basis? Can we sufficiently account for case mix (some types of patients and some procedures carry a higher risk of death)? However, what really bothers me is small numbers. According to the BBC article this rate will be based on a review of 2000 cases. Given that there are around 160 hospitals in England (depending on how you count them) this works out at an average of twelve and a half cases per hospital. That is a small number by which to judge a hospital.

Without knowing exactly how the review might work it is hard to know exactly how to criticize, so I will take a guess. I suspect it will start with some algorithm designed to identify the 2000 deaths with the highest chance of being avoidable based on what is contained within national data. After this each case will be reviewed to see how many of them were “avoidable” and this will then be translated into an avoidable death rate. If the algorithm was good enough to pick out death with a 50% chance of being avoidable (and I very much doubt that is the case) we would then be faced with basing hospital metrics on around six deaths per hospital. But of course that is an average. Even if all hospitals in the land had a long term avoidable death rate which would result in six cases using this method, sometimes they would have more and sometimes less just due to the role of chance. Under these circumstances we might very well expect the number of avoidable deaths per hospital to follow a Poisson distribution. The graph below shows the expected distribution of avoidable deaths in 160 hospitals from a Poisson distribution with a mean of six.

poisson

You can see from the graph that there is a big relative variation in the anticipated numbers of avoidable deaths. In particular, we would expect to see one hospital with thirteen avoidable deaths and two more with twelve. That is three hospitals with twice the national rate, purely by chance alone. Please note that this is what we expect on a typical year, and not what might possibly be seen when we get a lucky (or unlucky, depending on your perspective) roll of the dice. Considering these more extreme chance events we might expect to see a hospital with 15 avoidable deaths in around one year in every five.

It is quite clear from the graph that with small numbers like this the variation due to chance alone is large, so should we be publishing rankings based on chance? The Poisson distribution is not new and was famously applied, in 1898, to the Prussian cavalry describing the number of soldiers killed by being kicked by a horse each year in each of 14 cavalry corps between 1875 and 1894. So the statistics of small numbers of avoidable deaths, whether they come in the form of being kicked by a horse or occurring in an NHS hospital, have been known for over 100 years. One would hope that someone in the department of health is aware of these statistics and common sense will out before they are published. However, it would have been nice if someone had given Jeremy Hunt a statistical history lesson before he announced his plan to the world.

This entry was posted in Uncategorised. Bookmark the permalink. Both comments and trackbacks are currently closed.
  • The Cambridge Centre for Health Services Research (CCHSR) is a thriving collaboration between the University of Cambridge and RAND Europe. We aim to inform health policy and practice by conducting research and evaluation studies of organisation and delivery of healthcare, including safety, effectiveness, efficiency and patient experience.