Does pay for performance work or doesn’t it?

Too simple a question obviously. In our analysis of the first 18 months of Advancing Quality, a hospital pay for performance (P4P) scheme in the North West of England, we concluded that the scheme might have saved 890 lives (NEJM 2012; 367: 1821-8). However, when we looked at the effect two years later, the mortality advantage had disappeared (NEJM 2014; 371: 540-548).

So did that mean the scheme simply stopped working? Well, maybe. There were a number of reasons why it might have done, not least that it changed from a scheme where high performing hospitals were rewarded to one where low performing hospitals were penalised. However, it’s also possible that the apparent loss of effect was due to positive spillover effects onto control hospitals and conditions. Why might this have occurred?

Our first set of controls for hospitals in the North West of England were all other hospitals in England. After the initial apparent success of Advancing Quality (AQ), two other regions in England adopted a version of AQ in their hospitals, and in the second period (18 to 42 months) these regions had a larger reduction in mortality for the incentivized conditions compared to other English regions. So maybe the loss of difference between the North West hospitals and the rest of England was because the other hospitals had improved by adopting AQ.

Our second set of controls were patients in hospitals in the North West of England with non-incentivised conditions that weren’t related clinically to the incentivised ones. In our first analysis we found there were fewer deaths for the incentivised conditions than for these unrelated conditions, but again the difference appeared to be lost in the longer follow up period. When we looked at this in more detail, we found that mortality for these unrelated conditions had improved more in the North West than in the rest of England. Furthermore, when we looked at the overlap between consultants treating the incentivised and unincentivised conditions in the North West, we found that the improvement in mortality in non-incentivised conditions was greatest for conditions which happened to be treated by consultants also treating the incentivised conditions? So was there a general ‘quality improvement’ spillover on these firms?

This is all hard to sort out (and we can’t from our data). However, it raises the possibility of positive spillover effects from the scheme, which is just what one would want, but nonetheless may have made it look as if the scheme stopped working.

If there is a single message from this, it’s that pay for performance is a very complex intervention, and that the effects (positive and negative) almost certainly depend on the detail of implementation. The Agency for Healthcare Quality and Research (AHRQ) in the US is currently commissioning a series of definitive papers on P4P which will be published in 2015. These papers will give the most up to date guidance available on how to get benefits from P4P while avoiding perverse outcomes.

This entry was posted in Blog and tagged , , , . Group: . Bookmark the permalink. Both comments and trackbacks are currently closed.
  • The Cambridge Centre for Health Services Research (CCHSR) is a thriving collaboration between the University of Cambridge and RAND Europe. We aim to inform health policy and practice by conducting research and evaluation studies of organisation and delivery of healthcare, including safety, effectiveness, efficiency and patient experience.