Process or Outcomes for Quality Measurement?

Whether quality measurement should focus on process or outcomes measures is a perennial dilemma for health policy and health services researchers. Our recent research provides new insights to this old age dilemma.

How promptly cancer patients are diagnosed is a prevailing concern for UK population health and healthcare policy. This has led the development of several general practice indicators profiling diagnostic activity related to cancer diagnosis. These include the level of use of endoscopies and urgent referrals for suspected cancer, and also the % of cancer patients who were diagnosed as emergencies.

However, the statistical properties of those indicators are not adequately understood. New diagnosis of cancer are actually relatively rare with typically around 30 new cases in a general practice in a year. Given this, there are particular concerns about small numbers and the influence of chance. Therefore, in a recent paper from our team we examined this question empirically (1).

We a priori grouped the 13 diagnostic activity indicators into two major groups: process indicators and outcome indicators. In the context of our study, process indicators apply to patients that were investigated and most of whom were found not to have cancer (eg. endoscopy rates, screening coverage and urgent “two-week wait” referral rates). On the other hand, outcome indicators relate to a group of patient where all (or a high proportion) have cancer (eg. the proportion of patients diagnosed following an emergency presentation or the proportion of urgent referrals that resulted in a cancer diagnosis).

As expected outcome indicators (such as the % of cancer patients who are diagnosed as emergencies) apply to substantially fewer patients than process indicators. As a result the proportion of observed variation which can be attributed to chance is higher for outcome indicators than process ones. Consequently outcome indicators have low reliability, meaning that it is unwise to try to compare (rank) practices based on yearly data on outcome indicators. One way to overcome this problem is to aggregate data from unreliable indicators over multiple years in order to achieve a desired level of reliability. However, the good news is that practices can be reliably classified for most process indicators (such as the rate of use of endoscopies).

Another finding from this work is that there is substantial between-practice variation in all these indicators (beyond that expected by chance and from the variation in population demographics). There is benefit in trying to understand the causes of these differences with the aim of achieving better outcomes, as indeed we are doing currently in follow-on studies.

Our research provides an example of the importance of profiling the statistical properties of indicators in healthcare to optimise their use and interpretation.

 

  1. Abel, G., Saunders, C.L., Mendonca, S.C., Gildea, C., McPhail, S. and Lyratzopoulos, G., 2017. Variation and statistical reliability of publicly reported primary care diagnostic activity indicators for cancer: a cross-sectional ecological study of routine data. BMJ Qual Saf, pp.bmjqs-2017.
This entry was posted in Blog and tagged , . Bookmark the permalink. Both comments and trackbacks are currently closed.
  • The Cambridge Centre for Health Services Research (CCHSR) is a thriving collaboration between the University of Cambridge and RAND Europe. We aim to inform health policy and practice by conducting research and evaluation studies of organisation and delivery of healthcare, including safety, effectiveness, efficiency and patient experience.