ROC curves and the forgotten lesson of Pythagoras

You know the feeling – someone has said something at some point, but you can’t find the reference to back it up. Well according to Wikipedia, no specific attribution of Pythagoras’ theorem to Pythagoras exists in the surviving Greek literature from the five centuries after Pythagoras lived, let alone from the big man himself. So despite our best efforts we failed in our quest to cite a 2,500 year old document. But what has Pythagoras got to do with ROC curves anyway?

ROC curves have many uses, all of which fall into the theme of comparing a continuous (or quasi-continuous) measure with a dichotomous one. In one particular application, which is the subject of our paper published in PLOS One yesterday, we are concerned with selecting the smallest health change in an individual over two time points, which can be considered important (the Minimally Important Change, or MIC). This is often done by comparing changes in a continuous measure to a separate question on whether things have improved (or deteriorated). By plotting a ROC curve we display the sensitivity and specificity of the change in a continuous measure for detecting a dichotomous judgement about change. In the absence of any other motivations, it is often the case that sensitivity and specificity are valued equally and it is often argued that the point on the ROC curve closest to the top left hand corner is the cut-point that should be used.

If you stop and think about it, it should be obvious that to find the point closest to the top left hand corner, one can calculate the distance of each point to the corner and select the one where that distance is smallest. To calculate that distance you can apply Pythagoras’ theorem. So if (1-specificity) and (1-sensitivity) form two sides of a right angled triangle, the distance to the corner is the hypotenuse. In fact, this is so obvious that one methodological journal turned down our paper for just that reason.

Nevertheless, this is not how cut-points on ROC curves are being determined in practice. Instead, people find the point where the sum of (1-specificity) and (1-sensitivity) is smallest, rather than the sum of the squares suggested by Pythagoras’ theorem. It doesn’t even appear that this is bad practice being applied by only a handful of ill-informed researchers – just that this is the way everyone does it. Of course, in a well-behaved symmetric situation both techniques give the same result, but our universe is seldom well-behaved…

So if you are in the business of using ROC curves you should probably read the full gory techie details in our paper, and possibly install our Stata routine. And remember – even if something appears to you to be completely obvious, if you think everybody else is doing it wrong, it might be worth challenging them about it. You never know – you might be right.

Open Access Paper

Froud R, Abel, G. Using ROC curves to choose minimally important change thresholds when sensitivity and specificity are valued equally: the forgotten lesson of Pythagoras. Theoretical considerations and an example application of change in health status. PLOS One 2014; DOI: 10.1371/journal.pone.0114468

Stata Module

ROCMIC: Stata module to estimate minimally important change (MIC) thresholds for continuous clinical outcome measures using ROC curves – available from https://ideas.repec.org/c/boc/bocode/s457052.html or type “ssc install rocmic” at the stata prompt.

This entry was posted in Uncategorised. Bookmark the permalink. Both comments and trackbacks are currently closed.
  • The Cambridge Centre for Health Services Research (CCHSR) is a thriving collaboration between the University of Cambridge and RAND Europe. We aim to inform health policy and practice by conducting research and evaluation studies of organisation and delivery of healthcare, including safety, effectiveness, efficiency and patient experience.