How to write a referee report

Seriously cribbed from A Guide for the Young Economist.  This is how the majority (though not all) of economists do it, and when I’ve reviewed for other fields I’ve been complimented on the organization, so I don’t think you can go wrong using this format even if you’re not an economist.

The letter to the authors

Start with a paragraph called “Summary”.  There’s some disagreement if these are still needed or just waste time, but I think if you do the summary paragraph right, it can be useful to both the editor and the authors.  The summary should NOT just be a restatement of the abstract.  It should be a summary of what the actual paper is about, and not what the authors think it’s about.  So, for example, if it’s an experiment, you would have a sentence saying what you think the authors are trying to do (Ex. The authors explore the effect of salt water vs. fresh water on underwater basket weaving.)  Then you say briefly what they actually did.  (The authors did a randomized controlled experiment using a student population in which…)  You might end with a statement about how they extrapolate their claims to a broader issue.  Often the abstract doesn’t actually fit what the paper is about– it makes much larger claims about what happened.  The summary should be neutral and describe what the authors actually did.  This is helpful to the editor to know what the paper is actually about before they read through it, and helpful to the authors in case what your understanding of the paper is about is different than what they intended.  They can fix their writing to make the paper more clear.  I find it helpful to focus on the method section and tables only for this part.

Then create three sections:

Major:

Minor:

Minutia:

Major should include things that you think must be fixed before the paper is published.  If the paper is a reject, then this is where reasons for rejection would go.  If the paper is an R&R, these are the items that must be addressed for sure.

Minor:  This is where smaller questions go.  You might have things that need clarification, things that are incorrect, additional robustness checks that are not make-or-break but should be addressed, and so on.  It is helpful to include page numbers with these.

Minutia:  This is where all the typos, page proofing, etc. stuff should go.  You’re recognizing that they’re small mistakes that the authors will want to fix, but that they’re not big deals.  These definitely need page numbers.  (If one of your recommendations is “spellcheck” because there are multiple spelling problems, I would put that under minor, as opposed to saying “should it be here instead of hear on the first sentence of page 28?” which would be a minutia, but YMMV.)

Always be polite in your referee report, even if the paper is ridiculous.  Do not make a reject/R&R recommendation within the paper.  (Also:  as an editor I can say for certain that positive letters don’t always lead to R&R recommendations and negative letters don’t always mean the person recommended reject.  It’s insane how some people can say different things depending on the audience.)

Advice is generally that you do not have to spend as much time on reject papers as you do on R&R– some people will say just stating the major points is enough if you plan to reject.  As a reviewer I generally try to give advice for making the paper better should it go to another journal or should the editor disagree with my assessment, but sometimes a paper is just not publishable so it doesn’t matter if they never fix the typo in footnote 17 even if I found it.  I’ve found editing at a lower tier journal that reviewers tend to over recommend revise and resubmit (they’ll be like, “the paper says that correlation is causation, but if they could only get at causation, this would be a great paper, R&R”), and the explanations people give me are much more important than their actual recommendations.  My colleague who edits at a top journal says reviewers over reject (“this is the best paper I’ve ever read, Reject”), so the explanations are important.  When I was editing a top field journal, reviewers tended to get it “right” on average.

The letter to the editor

You will also generally have a letter to the editor.  I find the best editors letters provide a concise summary of the letter to the authors and possibly elaborate on the context of your comments– Basically reiterate the major points that led to your decision of reject, or explain what must be fixed before publishing.  If you don’t have much to say because it’s an obvious accept, use this space to fight for the paper.  You don’t have to be anonymous in the letter to the editor, so you can say more things that put it into context or explain what you’re not sure about because it’s not your area of expertise or what you are sure about because you are an expert.  If you’re not sure if it should be R&R or Reject, here is a good place to say so and explain why– what are the pros and cons?  These pros and cons should also be in your letter to the authors, but you can provide more context in your letter to the editor.  You can also put disclaimers in the letter to the editor like, “I didn’t realize when I accepted this paper that it was written by a former coauthor” or “I reviewed this paper for a top journal earlier and recommended it be sent to this journal instead.” Some dudes who read this blog think that there should never be anything said to the editor that isn’t in the letter to the authors, but I strongly disagree.  I appreciate the reiteration of the major points of the review (especially since some people don’t use must be fixed as their delineation between major and minor sections, but instead use difficulty of fixing etc.) and any context that I should know about (and I really don’t need to know about that typo on footnote 17 unless the paper is a revise and resubmit, but not everybody keeps those things to the minutia section).

Special topic:  Top journals

For top journals (for which I have not yet been an editor but have done a lot of reviewing), you may want to keep in mind the following points:

  1. Is it clean/well-done?  (This is the bare minimum)
  2. Is it Novel? (Doesn’t always have to be, but it helps a lot… though you can’t be too novel or it gets rejected because it’s “not economics” even if it actually is.  grrrr.)
  3. Does it make a major theoretical and/or empirical contribution to the field?  (Sometimes papers don’t need to add to empirics, but they do need to have a theory base even if not literally a theoretical model.)
  4. Is it Important/ of general interest? (This is highly subjective and where many of my papers strike out because it turns out they’re ahead of their time.  grrr.)

Update:  Here’s xykademiqz on the same topic for her science field.

Do you do a lot of referee reports?  How does your field handle them?

21 Responses to “How to write a referee report”

  1. Steph Says:

    I write 1-2 reports/year (I often enjoy them, is that weird?) The typical structure in my field is similar to yours; I split major/minor comments, but not everyone does. I think I’m going to start doing major/minor/minutia, as you’ve suggested here, because minor science issues aren’t the same thing as grammar/plot issues. Though I have put those things under “major” at times – there are a lot of people in my field who make truly terrible plotting or structural decisions, where it actually affects the interpretation of their results.

    Where is R&R in the hierarchy of responses in your field? Our journals take two roads: one has you make a soft evaluative statement at the end of that summary paragraph (“this should be published after corrections” or “can’t recommend publishing without major changes”), and the other has you pick from Accept (either “accept with minor revisions” or “accept with major revisions”) or Reject (“revise & resubmit” or “reject”). I’ve received an “R&R”/soft rejection under the first system, but only given one under the second system.

    Also, most papers in our field get only 1 referee, or 2 at most but it’s rare. This makes our refereeing process way weirder than other fields, and more prone to abuse.

    • Steph Says:

      Actually, that’s a lie, I’ve given an R&R under both. I just remembered the awful paper I reviewed from a senior researcher a couple years ago, where I sent it back with a nicer version of “almost nothing they wrote makes sense?!?”

    • nicoleandmaggie Says:

      I usually have at least one a month on top of editing (2 for one journal, 1 for another).

      In econ, we generally don’t give an R&R unless we think it’s going to be published. Different journals have different flavors and gradations of R&R but usually it’s Accept with minor revisions, Major revisions (R&R), Minor revisions (R&R), Reject. A couple journals have Accept as is/Reject with no choice to R&R, but I don’t like those.

      Papers generally get 2-4 reviewers (2-3 most common). Except journal of law and economics which just has 1. At one of the journals I did we had to get four! Which was hard because I was getting 10 papers/month. Thankfully that was only a one year thing (I finished out a 3 year term of someone who quit from overwork).

      • Steph Says:

        Gotcha, so R&R means something a bit different in our field then. I don’t know anyone who gets “Accept as is”, which might be why our tiers are a bit different.

        One report a month! And I can’t imagine that 4/paper journal, that had to be a nightmare for everybody.

        I think our field is much smaller than Econ, plus fewer referees/paper, so that might account for it. We have a few subfields that publish more frequently than mine (LPUs in the negative sense), so I imagine they referee more.

      • nicoleandmaggie Says:

        It’s only a couple weird journals that do the accept as is, and it’s relatively new. I think they’re experimenting. I do not like reviewing for them.

        I think I’m a more popular reviewer than a lot of people because I’m something of a dilettante in terms of subject matter, I have a good pedigree, but am not so impressive that I must be too busy to review.

        Some months I get as many as 6 requests (and sometimes feel like I have to accept 4). Sigh. I am going to have to cut down because I’m also editing 4/month now.

  2. Michael Nitabach Says:

    “some dudes who read this blogge” 😹😹😹

    This is a great post, and I particularly agree with providing the summary paragraph (can be just a few sentences) at the start of the review. This establishes your credibility as reviewer to both the editors & authors that you read & understood the paper.

  3. CG Says:

    I do 10-12 per year and I generally enjoy doing them. I like the summary paragraph idea and should do a better job of that. I think I usually do it but maybe I should put more effort into it. I definitely do the major (overall) comments and the more specific comments approach. I try to imagine that the author is a Ph.D. student or young assistant professor and write my report accordingly. I probably occasionally let snark or superiority creep in but try hard not to. The exception to this is if someone says something like “everything that came before me (which often includes my work) is crap and I’ve got it all figured out.” Rarely is that statement accurate. I only rarely write a separate letter to the editor, but maybe I should start doing that more often.

    I’m doing two revisions of my own right now and on one of them one reviewer identified a problem and proposed two solutions, one pretty involved and one to just get rid of the section if it wasn’t important to the paper. I really appreciated that since I probably would have gone down a rabbit hole trying to fix the problem when I could just make it go away (research-design-wise it was fine to do that). Another reviewer said we had made a serious oversight not including race when it is literally IN OUR MODEL. That makes me mad and makes me suspect this person did not read our paper as carefully as he or she should have.

    One of my life goals is being an associate journal editor (maybe even a journal editor if I like being an associate).

  4. Fiona McQuarrie Says:

    I use the framework that you describe, and it’s really helpful to me as a reviewer in organizing my thoughts. An additional suggestion is to number the points within each section. That makes it easier for the editor to write the letter to the author(s) because then they can say “As reviewer X says in (1)” instead of “As reviewer X says in their discussion of methodology and your choice of sample”.

  5. xykademiqz Says:

    Great post! In my field, there is usually no separate letter to the editor. There’s a box for comments to the editor where you can put stuff you don’t want forwarded to the authors, but I’ve never seen it be a full letter and I seldom use it myself. Also, all referee forms come with some sort of questionnaire or a drop-down menu where you can select numerical scores for various aspects of paper’s merit (quality of writing, originality, length, figure quality, perceived impact on the field, etc.), as well as select a specific recommendation for editorial action. All this is, of course, in addition to a full report and, if you want, a bit of info for editor’s eyes only.

    Fun anecdote: In my role as associate editor, I received comments from a referee saying the paper was really boring because there was nothing creative about what the authors have done; the referee was not wrong. However, the report he/she wrote was professional and quite neutral in tone, and you wouldn’t have inferred the truly negative opinion of the work the referee held.

    I have also occasionally used the comments-to-the-editor box to share my impressions in a more succinct and blunt / less sugarcoated form than what gets forwarded to the authors.

    • nicoleandmaggie Says:

      Some of our journals have that same checkbox system — which often has some boxes that don’t make sense because they’re really for hard science.

    • Lisa Says:

      We also only rarely use the box for comments to the editor. I’m not sure I’ve ever used it – the one time I had serious concerns about a manuscript, I called the editor instead. I was hesitant to put my concerns in writing, but I knew the editor somewhat from previous interactions and ended up very impressed with how they handled my concerns. I was able to keep the review professional and neutral, but still get my concerns addressed.

      Also, I’ve become very picky about which papers I will agree to review and now only say yes if there’s a really good reason (I know the authors and know they usually do good work, I’m one of few people with the right expertise to review, etc.). I’ve really cut it down from 1-3 a month to just a few a year because I’m busy with other administrative stuff. I can’t even fathom taking on the unpaid work of being an associate editor, although I’m immensely grateful that others are willing to do this. I’m sure there are tangible benefits to being intimately involved in publishing. Perhaps it comes down to personal preference at my career stage – I’m sure there are many who couldn’t imagine ever wanting to take on my administrative role.

  6. Debbie M Says:

    As a non-scientist, I just want to say:

    1. In response to “Is it clean/well-done?” yes! Or in my brain, “is this actual science?”

    2. Thank you to all of you who help to figure things out and help communication and spread around the best new info!

    3. My favorite science is a) where you don’t already know the answer before you start and b) trying to replicate someone else’s non-intuitive results. Well, okay, really my favorite science is where you do already know the answer before you start, but then find out you are wrong (and go with it instead of trying to cover it up).

  7. undine Says:

    Great post! We don’t have “letter to the editor” in my discipline, although some forms ask for separate comments for the editor.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: