Seriously cribbed from A Guide for the Young Economist. This is how the majority (though not all) of economists do it, and when I’ve reviewed for other fields I’ve been complimented on the organization, so I don’t think you can go wrong using this format even if you’re not an economist.
The letter to the authors
Start with a paragraph called “Summary”. There’s some disagreement if these are still needed or just waste time, but I think if you do the summary paragraph right, it can be useful to both the editor and the authors. The summary should NOT just be a restatement of the abstract. It should be a summary of what the actual paper is about, and not what the authors think it’s about. So, for example, if it’s an experiment, you would have a sentence saying what you think the authors are trying to do (Ex. The authors explore the effect of salt water vs. fresh water on underwater basket weaving.) Then you say briefly what they actually did. (The authors did a randomized controlled experiment using a student population in which…) You might end with a statement about how they extrapolate their claims to a broader issue. Often the abstract doesn’t actually fit what the paper is about– it makes much larger claims about what happened. The summary should be neutral and describe what the authors actually did. This is helpful to the editor to know what the paper is actually about before they read through it, and helpful to the authors in case what your understanding of the paper is about is different than what they intended. They can fix their writing to make the paper more clear. I find it helpful to focus on the method section and tables only for this part.
Then create three sections:
Major should include things that you think must be fixed before the paper is published. If the paper is a reject, then this is where reasons for rejection would go. If the paper is an R&R, these are the items that must be addressed for sure.
Minor: This is where smaller questions go. You might have things that need clarification, things that are incorrect, additional robustness checks that are not make-or-break but should be addressed, and so on. It is helpful to include page numbers with these.
Minutia: This is where all the typos, page proofing, etc. stuff should go. You’re recognizing that they’re small mistakes that the authors will want to fix, but that they’re not big deals. These definitely need page numbers. (If one of your recommendations is “spellcheck” because there are multiple spelling problems, I would put that under minor, as opposed to saying “should it be here instead of hear on the first sentence of page 28?” which would be a minutia, but YMMV.)
Always be polite in your referee report, even if the paper is ridiculous. Do not make a reject/R&R recommendation within the paper. (Also: as an editor I can say for certain that positive letters don’t always lead to R&R recommendations and negative letters don’t always mean the person recommended reject. It’s insane how some people can say different things depending on the audience.)
Advice is generally that you do not have to spend as much time on reject papers as you do on R&R– some people will say just stating the major points is enough if you plan to reject. As a reviewer I generally try to give advice for making the paper better should it go to another journal or should the editor disagree with my assessment, but sometimes a paper is just not publishable so it doesn’t matter if they never fix the typo in footnote 17 even if I found it. I’ve found editing at a lower tier journal that reviewers tend to over recommend revise and resubmit (they’ll be like, “the paper says that correlation is causation, but if they could only get at causation, this would be a great paper, R&R”), and the explanations people give me are much more important than their actual recommendations. My colleague who edits at a top journal says reviewers over reject (“this is the best paper I’ve ever read, Reject”), so the explanations are important. When I was editing a top field journal, reviewers tended to get it “right” on average.
The letter to the editor
You will also generally have a letter to the editor. I find the best editors letters provide a concise summary of the letter to the authors and possibly elaborate on the context of your comments– Basically reiterate the major points that led to your decision of reject, or explain what must be fixed before publishing. If you don’t have much to say because it’s an obvious accept, use this space to fight for the paper. You don’t have to be anonymous in the letter to the editor, so you can say more things that put it into context or explain what you’re not sure about because it’s not your area of expertise or what you are sure about because you are an expert. If you’re not sure if it should be R&R or Reject, here is a good place to say so and explain why– what are the pros and cons? These pros and cons should also be in your letter to the authors, but you can provide more context in your letter to the editor. You can also put disclaimers in the letter to the editor like, “I didn’t realize when I accepted this paper that it was written by a former coauthor” or “I reviewed this paper for a top journal earlier and recommended it be sent to this journal instead.” Some dudes who read this blog think that there should never be anything said to the editor that isn’t in the letter to the authors, but I strongly disagree. I appreciate the reiteration of the major points of the review (especially since some people don’t use must be fixed as their delineation between major and minor sections, but instead use difficulty of fixing etc.) and any context that I should know about (and I really don’t need to know about that typo on footnote 17 unless the paper is a revise and resubmit, but not everybody keeps those things to the minutia section).
Special topic: Top journals
For top journals (for which I have not yet been an editor but have done a lot of reviewing), you may want to keep in mind the following points:
- Is it clean/well-done? (This is the bare minimum)
- Is it Novel? (Doesn’t always have to be, but it helps a lot… though you can’t be too novel or it gets rejected because it’s “not economics” even if it actually is. grrrr.)
- Does it make a major theoretical and/or empirical contribution to the field? (Sometimes papers don’t need to add to empirics, but they do need to have a theory base even if not literally a theoretical model.)
- Is it Important/ of general interest? (This is highly subjective and where many of my papers strike out because it turns out they’re ahead of their time. grrr.)
Update: Here’s xykademiqz on the same topic for her science field.
Do you do a lot of referee reports? How does your field handle them?