What does “good” really mean in patient experience surveys?

Evaluating our experiences can be tricky. I recently completed a two-day training course: the trainer was great, the content engaging and helpful, and I dutifully went down the evaluation form giving glowing reviews all round. In the circumstances, my assessment felt very genuine. Of course, there is the fact that the trainer was still standing at the front of the room, and I admit that the thought of handing back mean or negative comments to them was an unpleasant one. I don’t think it changed my evaluation in this case, but if the trainer hadn’t been so good, would I have dared to state this on the form? I’d be less inhibited in criticising a restaurant or a hotel, I think: it feels less personal, more like a object – hence the vigorous venting of spleen you see plastered all over Tripadvisor. In fact, the most dire Tripadvisor reviews become news stories in their own right, as people let rip at the alleged incompetence, filth, and abuse paying customers are apparently expected to put up with.

Of course, some things we are asked to evaluate are rather more complex than a meal. Medical care, for example. Thinking back to the last time you saw your GP, how good were they at….? The national GP Patient Survey would ask you to evaluate how well your GP did at listening to you, at explaining tests and treatments, and other interpersonal aspects of care. And these answers matter: the Care Quality Commission’s ‘intelligent monitoring’ of GP practices relies heavily on these survey responses, with key indicators including the percentage of respondents who stated that the GP was good or very good at treating them with care and concern, and involving them in decisions about their care. Such interpretations assume that “good” means, you know – “good”. However, our latest research suggests that might not be the case.

To investigate this issue, we asked patients coming in to see their GP if they would consent to the consultation being filmed, and complete a questionnaire immediately afterwards. We then took the film of the consultation out to the patients at home, and watched it again with them, asking them to talk through how they chose their response options. Really, what we were asking was: why did they tick the box they did? Our analyses of their interview accounts in comparison to their responses on the questionnaire identified a clear gap: comments made during interviews did not always express the same sentiment as their responses to the questionnaire.  Where questionnaire responses indicated that interpersonal skills were ‘very good’, 84.6% of interview statements concerning that item could be classified as positive. However, where patients rated interpersonal skills as ‘good’, only 41.9% of interview statements were classified as positive, and 18.9% as negative. Our conclusions? Positive responses on patient experience questionnaires can mask important negative experiences which patients describe in subsequent interviews. So, what does this mean in practice? For clinicians, the simple message is: don’t be complacent following the receipt of ‘good’ feedback from surveys, as ‘good’ may not be good enough. At a wider level, interpreting patient experience scores for use in public reporting should be done with caution: whilst the use of relative rankings is common, questions remain about the point at which differences between providers becomes of concern, and the practice of classifying ‘very good’ and ‘good’ response options together as indicative of positive experiences of care.

For full details, see:

Burt J, Newbould J, Abel G, Elliott MN, Beckwith J, Llanwarne N, Elmore N, Davey A, Gibbons C, Campbell J, Roland M. Investigating the meaning of ‘good’ or ‘very good’ patient evaluations of care in English general practice: a mixed methods study. BMJ Open 2017

This report presents independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research Programme (Grant Reference Number RP-PG-0608-10050). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. The researchers confirm their independence from the study funders, the National Institute for Health Research.

This entry was posted in Blog and tagged , . Group: . Bookmark the permalink. Both comments and trackbacks are currently closed.

One Comment