Let not thy will roar, when thy power can but whisper

The title quote is from Thomas Fuller (1654 – 1734), English churchman and historian

I recently heard someone say that there is no point in doing sample size or power calculations for observational studies with routine data as you have the data that you have and it tells you what it tells you. The fact that this is coming at the start of a blog is probably enough to let you know that I think this is utter hogwash. But why? After all, if there is nothing you can do about your sample size, what purpose is a power calculation going to serve? Before we get to the solution let’s think about the problem.

The power of a statistical test is the probability of finding a statistically significant association when one exists and it depends on three things:

  1. The strength of the association
  2. The variability of the data
  3. The sample size

Now we don’t normally know the strength of an association before performing a test, as that is what we are trying to find out. So we often estimate the power we would have to detect an important or interesting association, and what this will be will very much depend on the context. We will always have more power to detect a larger association than a smaller one. There is normally little that can be done about the variability of the data, but when data is more variable there will be less power. In other words, it is harder to detect differences between groups where there is large variation within groups. With experimental studies or studies where you determine the volume of data collection, sample size is normally the variable in your control. By increasing the sample size power will go up. As such, in a well-designed study where we control data collection power should be adequate. However, when using routine data (as we often do in CCHSR) you’re often not in control and so we may well not have the power we might like. So what is the problem with not having enough power?

Consider the situation where 100 studies are performed to detect the same effect size. This may be 100 different researchers around the globe trying to answer the same question with different data sets, or equally one researcher trying to answer 100 different questions with one data set, or even 100 different researchers trying to answer 100 different questions – but all with the same true effect size. (If you think of how many people do research this is not particularly difficult to imagine). The figures below show two simulations of just such a situation. The blue line shows the true effect size, in this example a difference between groups of 2. In the first figure, the theoretical power is 30% and we see that around 30% of estimates are statistically significant – these are coloured red. Because power is low, confidence intervals are large and thus all the detected (significant) effects are larger than the true effect. In fact the red line shows the mean of the significant results which is 3.8, nearly twice the true effect size. The second figure shows the same situation when we instead have 80% power. In this simulation as expected around 80% of estimates are statistically significant. Whilst we do still see that the significant results are on average higher than the true effect (mean of significant estimates =2.3), this bias is much reduced compared to the 30% power situation.

So does power matter when using routine data and is there a point in performing a sample size/power calculation? I believe there is. If we all start doing underpowered analyses just because we can we will end up overestimating the importance of those effects we do find. Of course, we also have the problem that we may dismiss true effects as insignificant too easily; proper interpretation of confidence intervals should not make this a problem, but that is the subject for a different rant blog!

Figure 1: 30% power

Figure 1: 30% power

 

Figure 2: 80% power

Figure 2: 80% power

This entry was posted in Uncategorised. Bookmark the permalink. Both comments and trackbacks are currently closed.
  • The Cambridge Centre for Health Services Research (CCHSR) is a thriving collaboration between the University of Cambridge and RAND Europe. We aim to inform health policy and practice by conducting research and evaluation studies of organisation and delivery of healthcare, including safety, effectiveness, efficiency and patient experience.