Keeping Cochrane reviews up-to-date

The Cochrane Collaboration is an organisation that conducts systematic reviews of healthcare interventions.  The idea is that they are updated periodically (every 2 years) so that they provide the best evidence on what works and what doesn’t.

After 21 years of existence, the Cochrane Library now holds over 6000 reviews by some 20,000 volunteer authors.  Keeping them all up to date is becoming a huge headache: only 20-30% of reviews are currently classed as ‘up to date’, risking the credibility of the entire organisation.

Therefore a workshop was convened at McMaster University in Hamilton, Ontario this month, to discuss a broad range of issues ranging from practical editorial issues such as what to do with authors who ‘squat’ on their reviews not letting go of authorship but without updating them, to how to decide whether a review needs updating in the first place.

My contribution was towards this latter aspect, where value of information analysis can be used to predict the expected benefit to a population of updating a review.  The logic is as follows:

  • Uncertainties in health care mean there is a probability of harm associated with any decision (i.e. there is a probability that using treatment X is either going to not work thus wasting resources which could have benefited another patient, or fail to help them when treatment Y could have done)
  • The purpose of a systematic review is to reduce uncertainties in health care
  • This equates to reducing the probability of harm
  • Value of information analysis translates this reduction in probability of harm into an expected quantity of benefit for a population (e.g. expected number of patients who get the ‘correct’ treatment or expected QALY gain)

To apply this to updating a Cochrane review requires firstly quantifying the (prior) ‘expected harm’ associated with uncertainty in the current evidence base: the results of the existing review provide the information needed.  Secondly, a very quick automated scan of the literature is done to predict the number of new studies that may be included in an update of the review.

The results of updating the systematic review can then be predicted, and the ‘expected harm’ from the updated review calculated (which should be lower than the prior!).  The difference between the prior and (‘predicted posterior’) expected harm is the expected benefit to a population from updating a review.

The number of new studies identified in the scan gives an indication as to the likely workload to update a review in terms of number of day’s work.

Dividing the cost (days of work) by the benefit gives an estimate of the cost-effectiveness of updating a Cochrane review.  Reviews can then be ranked in order of increasing ‘days of effort per patient expected to benefit’ to give a prioritised list.

Ultimately this could become part of a semi-automated suite of tools to help Cochrane editors decide which reviews to tackle first.  Watch out for a short summary of this work in the next issue of Cochrane Methods, updates to the Cochrane handbook as well as a set of formal papers coming out of the workshop.

This entry was posted in Blog and tagged , , . Group: . Bookmark the permalink. Both comments and trackbacks are currently closed.
  • The Cambridge Centre for Health Services Research (CCHSR) is a thriving collaboration between the University of Cambridge and RAND Europe. We aim to inform health policy and practice by conducting research and evaluation studies of organisation and delivery of healthcare, including safety, effectiveness, efficiency and patient experience.