Reviewing a paper? Here’s what I do!

Following on from the storming success of my random thoughts on revising a paper, I thought I would treat everyone to a Part II: what do I do when reviewing a paper? To set the scene, try reading this classic paper by Peters and Ceci, in which they took recently published papers, added fictitious author names, and re-submitted them to the same twelve journals. The results were not pretty. Nine of the 12 papers slipped straight through undetected. Under peer review, 16 of 18 referees recommended against publication, mostly because of “serious methodological flaws.” And here lies the much-touted problem with peer review: it can be biased, sloppy, poor quality, unhelpful, erratic and even spiteful. Shudder to think we should ever fall into that trap. Deep breaths, and off we go.

1. Make sure you can do it

Common sense really, but first off I always ask myself – do I actually feel competent to comment on this? Those “interests/expertise” lists you get asked to tick on journal websites don’t necessarily give editors that much of a clue of what you are good at, and I have been asked to review some pretty off piste topics. In truth, this does not necessarily mean you need to be an expert on every aspect of the paper. Just make sure you feel comfortable in the general field or methods used before you accept the task. Making sure you can do it also means checking you have enough time to complete the review by the deadline, which is only polite for the authors who are waiting (although an editor has taken pity and granted me an extension following the “a dog ate my arm and I can’t type” incident. Make sure your excuses sound plausible, I always say…).

2. Do it properly

We all know the utter wretchedness caused by your paper receiving an ill-thought out paragraph or two of harsh words and no constructive suggestions. Reviewing a paper means just that – reviewing it. Carefully. Thoughtfully. Helpfully. All journals will provide a checklist for the different types of paper. Use this as your starting point. If they have recommended headings to structure your review, use these. Usually they cover brief notes on the suitability for the journal, abstract, introduction, methods, results, conclusions, references, tables and figures, readability, general comments, and so on (and on). These are pretty helpful, and do ensure you are likely to provide a reasonably comprehensive assessment. So, how do I go about getting mine down?

  • I read the paper, and put it aside for a bit. I read it again.
  • In a word document, I note down my general impressions – usually any major concerns that have leapt out at me. These are the headline thoughts I want to feed back to the editor.
  • I then go through the paper with a fine tooth comb. It is amazing how any paper gives up its dark secrets the more you study it. I write notes on every area of the paper, and I usually end up with around two pages of comment (in dire cases, many more than this). However, some papers are so awful there is no point going through finding typos when they just need to start all over again.
  • Whilst I am doing this, I also do a quick search for relevant and recent literature. However much I feel comfortable in a subject, I always want to check to see what may have been added which should have been referenced in a paper, and I have a quick read of anything which looks helpful.
  • I may also want to double-check specific methods used, data sources and a whole host of other things. The point here is that I never simply read and comment on the paper – I always pull in a range of information to help me in my review.
  • Once I am happy with my review, I transfer it to the journal website (or follow whatever other instructions they have given me). It can be nerve wracking hovering over that button with the stark choices “accept it is fab/sort of accept it is OK/reject it is terrible etc.” In particular, how should you decide between minor or major revision? Normally it is pretty obvious, but do also remember the editor is there to make the final judgement call, and they will rely much more on the detail of your comments to do so. If there is the option to give confidential comments to the editor, I always do, and this is often helpful to flesh out the reasons for your recommendation (or your uncertainty about which box it falls into).

And how much time does this take? I have no idea what editor’s expectations are here. I usually spend at least two hours once I sit down to the task, sometimes a lot more – and this is after reading the paper a couple of times previously. It doesn’t sound like much, really, for the years of work someone has often put in (or so I tell myself when I am feeling swamped with other things to do).

3. Put yourself in their shoes

If you remember nothing else whilst completing a review, just think how the authors will feel when they get your report. Many of you will have heard of the “sandwich approach” to delivering feedback: nice, not so nice, nice. However bad a paper is, I always try to be constructive; a long list of criticisms with no suggestions for what the authors can change is not helpful. Before I submit, my golden rule is to visualise the author, sitting at their desk, reading your words. How do they look? Crushed? Hysterical? OK? Could you deliver your review face to face, even if this meant being very brave and firm? If not, think about going back and toning things down. You do have to be honest, but you do not have to be rude.

4. Give yourself a pat on the back

Peer review is what makes the academic world go round, whether we like it or not. Peer reviewing is an important role for all members of that community: we can’t expect to publish our own papers without helping in the publication of other people’s. The skills learnt in assessing other papers are also super useful when it comes to writing your own. My standard question from project design through to writing up is “what would a reviewer say?” Hopefully, of course: “Oh my god! This is AMAZING!”


Need something more sensible to read on the subject?

  • The BMJ has lots of resources for reviewers, including links to their detailed guidance for reviewers (cheerfully, they note that “Peer review must often seem like a thankless task”. Thanks, guys)
  • Sense about Science published a useful review on how the public could make better sense of peer review, called “Peer review and the acceptance of new scientific ideas”. Although aimed at demystifying the peer review process, it has some useful examples of the nature of peer review and the kinds of comments reviewers make (read Section 1, page 7 onwards). There’s also an up-to-date timeline of everything they have done on peer review since then, with loads of useful links. In their original report, they state the peer review process can be “almost as complicated and unpredictable as the birth of a baby”. Brilliant.
  • The Committee on Publication Ethics recently published (March 2013) really useful guidelines on being an upstanding peer reviewer.
  • Go read about the fascinating (and very important) world of “post-publication” peer review on Dorothy Bishop’s blog
  • Matt Ayres, who is a Biological Sciences Prof at Dartmouth College in the US, has some sensible advice based on his experience in PDF form. And do check out his home page and see if you can guess which one he is in the picture – its brilliant.
  • For a rather different take on peer review (sample line: “The truth is that peer review is largely hokum”), read Nigel Hawkes’ piece in the Independent
  • And so as not to end on a negative note, here’s another really good blog on doing a peer review from Ethan Siegel (@StartsWithABang), with even more fabulous pictures. And yes, it is from the world of astronomy, but it is nice to see what they get up in other fields sometimes…


This entry was posted in Blog and tagged , . Group: . Bookmark the permalink. Both comments and trackbacks are currently closed.

One Comment