Significantly significant?

A complaint often levelled at medical research is that far more focus is given to statistical significance than clinical significance. In fact I saw a tweet a few weeks ago deploring the situation. This made me think why this is the case. The short answer is that we have some fairly well established thinking about when something is considered statistically significant (although the strength of evidence required to be convincing may depend on the application). However, the same cannot be said of clinical significance as it takes a degree of clinical judgement. I could end this blog here and leave it at the feet of my clinical colleagues, but there is more that the data analyst can do.

One of the most important things is to make sure that effects of continuous variables (from regression models) are given for meaningful quantities. As an example it may be that a model output suggests the odds ratio for a one-year change in age is 1.05. This appears at first glance to be a small effect. However, if we are looking at something which affects people in middle and old age it might be more appropriate to consider the odds ratio for a 20-year change in age, i.e. comparing a 70 year old with a 50 year old. Recasting the same model output would give the odds ratio for a 20-year change in age as 2.65, clearly a substantial effect.

Another issue with clinical significance is it depends at what level you are considering the effects. A small effect which applies to a lot of people may add up to sizable effects when summed over the whole population. One thing that can be done in such circumstances is to use a regression model to predict the population level outcome if everyone’s exposure was the same or if variation in outcome across a characteristic was removed. We used this technique to good effect in this paper where we considered socio-demographic inequalities in early/late stage cancer diagnosis for 10 common cancers. We suggested that elimination of these inequalities could decrease the number of people with cancer diagnosed at advanced stage in England by around 5600 annually. Of course such illustrations should be considered with caution as causality is implied and there may be extrapolation involved.

Running a model to show statistical significance is easy. Presenting the results in a meaningful way, where the clinical significance (or lack thereof) is clear, can take a little more work but is well worth it.

This entry was posted in Stats and tagged , . Group: . Bookmark the permalink. Both comments and trackbacks are currently closed.
  • The Cambridge Centre for Health Services Research (CCHSR) is a thriving collaboration between the University of Cambridge and RAND Europe. We aim to inform health policy and practice by conducting research and evaluation studies of organisation and delivery of healthcare, including safety, effectiveness, efficiency and patient experience.