Thoughts on academic journal editing again

  • I’ve gotten a couple of replies from authors I’ve rejected from the journal I’m an AE at thanking me and the reviewers for our excellent and helpful comments.  I didn’t know that was a thing people did.  (The last journal I was AE at the only replies I got were asking me to reconsider my decision!  I’ve only gotten one of those so far with this journal and it was a reject-and-resubmit in which there was a fatal flaw in the paper, none of the reviewers wanted to see a revision, and fixing it was doable but would create an entirely different paper and could fundamentally change the results… my letter specifically said that changing all the minor things was not enough, and yet, they sent back a response in which they fixed all the minor things and not the major and asked me to reconsider.  Don’t be those people.)  I think as editor I think a bit more fondly of the authors who have sent the thank yous than I did before they sent it… like it’s a little weird, but also, the reviewers and I *did* send super helpful comments in each of those cases.  So maybe I’ll start doing that, but only when the comments are helpful(!)  Though I’m pretty sure Larry Katz has already formed an opinion of me (the QJE provides the most helpful comments of any journal, in my experience).
  • I also didn’t realize until very recently that the journal I’m currently at doesn’t automatically send the final results to the reviewers, just the authors.  The last journal I was at did.  (They’re different systems with different pluses and minuses.)  So I went back and sent those out for everything I’d processed since December.  All the grad students I used replied with thank-you emails which I wasn’t expecting.
  • If you are an editor and you’re planning to send out to 3-4 reviewers, keep a list of graduate students and early assistant professors as you come into contact with them as well as their areas of expertise.  Reviewing helps them because generally they don’t have many journals listed on their cvs and if you send them the other reviews they learn about the reviewing process (also they’re more likely to say yes) and they tend to take it seriously.  For the most part they’ve done an excellent job in terms of comments.  I try to only have one or two though and have the remaining people with more direct experience with the topic of the paper.  I call my list, “Victims”.
  • My undergraduate advisor sent me a paper to review last month, so I sent her one this month.  She hasn’t accepted it yet.  I’ve started adding other editors to my victims list (not Larry Katz though!)
  • At the top field journal I edited for, people got reject/R&R about right.  At my current journal which is a lower tier journal, they over R&R by a LOT (we’re supposed to only R&R about 10% give or take– it’s not a hard and fast number, but if I just listened to reviewers I’d be R&Ring like 75% even when they note fatal flaws that might not be fixable and could fundamentally change the paper if fixable).  One of my colleagues who edits for a top journal says people reject too much.   I suspect there’s a nonlinear function that intersects at the top field journal level.  I know before I took this job I only recommended reject at this level of journal if I thought the paper was not publishable anywhere.
  • When I send to people with mixed experiences editing (so some with no experience and some at the journal I’m at, for example), I will often get nearly identical comments with completely different recommendations.  The words you use are much more helpful at the journal I’m currently at.
  • I like clearing my slate for editing as quickly as possible.  I worry that I’m getting sent more papers because of this.  :/
  • One thing I didn’t realize before I started editing is how draining just making the decisions in the process is.  Fortunately for this journal I don’t have to decide on desk rejects, but I do still have to decide who to send out to and then after I get their reports, how to translate them to whether or not the paper should be rejected.
  • I find it really helpful when reviewers explain in plain terms in their letter to the editor what the main concerns with the paper is (or alternately, why they think it should be given an R&R).  It’s much harder when reviewers only send the comments to the authors, because it’s harder to get a handle on which bits are the ones responsible for a decision of Reject (or alternately, why it should be revised if they don’t spend some time explaining that in the author letter).

How to write a referee report

Seriously cribbed from A Guide for the Young Economist.  This is how the majority (though not all) of economists do it, and when I’ve reviewed for other fields I’ve been complimented on the organization, so I don’t think you can go wrong using this format even if you’re not an economist.

The letter to the authors

Start with a paragraph called “Summary”.  There’s some disagreement if these are still needed or just waste time, but I think if you do the summary paragraph right, it can be useful to both the editor and the authors.  The summary should NOT just be a restatement of the abstract.  It should be a summary of what the actual paper is about, and not what the authors think it’s about.  So, for example, if it’s an experiment, you would have a sentence saying what you think the authors are trying to do (Ex. The authors explore the effect of salt water vs. fresh water on underwater basket weaving.)  Then you say briefly what they actually did.  (The authors did a randomized controlled experiment using a student population in which…)  You might end with a statement about how they extrapolate their claims to a broader issue.  Often the abstract doesn’t actually fit what the paper is about– it makes much larger claims about what happened.  The summary should be neutral and describe what the authors actually did.  This is helpful to the editor to know what the paper is actually about before they read through it, and helpful to the authors in case what your understanding of the paper is about is different than what they intended.  They can fix their writing to make the paper more clear.  I find it helpful to focus on the method section and tables only for this part.

Then create three sections:

Major:

Minor:

Minutia:

Major should include things that you think must be fixed before the paper is published.  If the paper is a reject, then this is where reasons for rejection would go.  If the paper is an R&R, these are the items that must be addressed for sure.

Minor:  This is where smaller questions go.  You might have things that need clarification, things that are incorrect, additional robustness checks that are not make-or-break but should be addressed, and so on.  It is helpful to include page numbers with these.

Minutia:  This is where all the typos, page proofing, etc. stuff should go.  You’re recognizing that they’re small mistakes that the authors will want to fix, but that they’re not big deals.  These definitely need page numbers.  (If one of your recommendations is “spellcheck” because there are multiple spelling problems, I would put that under minor, as opposed to saying “should it be here instead of hear on the first sentence of page 28?” which would be a minutia, but YMMV.)

Always be polite in your referee report, even if the paper is ridiculous.  Do not make a reject/R&R recommendation within the paper.  (Also:  as an editor I can say for certain that positive letters don’t always lead to R&R recommendations and negative letters don’t always mean the person recommended reject.  It’s insane how some people can say different things depending on the audience.)

Advice is generally that you do not have to spend as much time on reject papers as you do on R&R– some people will say just stating the major points is enough if you plan to reject.  As a reviewer I generally try to give advice for making the paper better should it go to another journal or should the editor disagree with my assessment, but sometimes a paper is just not publishable so it doesn’t matter if they never fix the typo in footnote 17 even if I found it.  I’ve found editing at a lower tier journal that reviewers tend to over recommend revise and resubmit (they’ll be like, “the paper says that correlation is causation, but if they could only get at causation, this would be a great paper, R&R”), and the explanations people give me are much more important than their actual recommendations.  My colleague who edits at a top journal says reviewers over reject (“this is the best paper I’ve ever read, Reject”), so the explanations are important.  When I was editing a top field journal, reviewers tended to get it “right” on average.

The letter to the editor

You will also generally have a letter to the editor.  I find the best editors letters provide a concise summary of the letter to the authors and possibly elaborate on the context of your comments– Basically reiterate the major points that led to your decision of reject, or explain what must be fixed before publishing.  If you don’t have much to say because it’s an obvious accept, use this space to fight for the paper.  You don’t have to be anonymous in the letter to the editor, so you can say more things that put it into context or explain what you’re not sure about because it’s not your area of expertise or what you are sure about because you are an expert.  If you’re not sure if it should be R&R or Reject, here is a good place to say so and explain why– what are the pros and cons?  These pros and cons should also be in your letter to the authors, but you can provide more context in your letter to the editor.  You can also put disclaimers in the letter to the editor like, “I didn’t realize when I accepted this paper that it was written by a former coauthor” or “I reviewed this paper for a top journal earlier and recommended it be sent to this journal instead.” Some dudes who read this blog think that there should never be anything said to the editor that isn’t in the letter to the authors, but I strongly disagree.  I appreciate the reiteration of the major points of the review (especially since some people don’t use must be fixed as their delineation between major and minor sections, but instead use difficulty of fixing etc.) and any context that I should know about (and I really don’t need to know about that typo on footnote 17 unless the paper is a revise and resubmit, but not everybody keeps those things to the minutia section).

Special topic:  Top journals

For top journals (for which I have not yet been an editor but have done a lot of reviewing), you may want to keep in mind the following points:

  1. Is it clean/well-done?  (This is the bare minimum)
  2. Is it Novel? (Doesn’t always have to be, but it helps a lot… though you can’t be too novel or it gets rejected because it’s “not economics” even if it actually is.  grrrr.)
  3. Does it make a major theoretical and/or empirical contribution to the field?  (Sometimes papers don’t need to add to empirics, but they do need to have a theory base even if not literally a theoretical model.)
  4. Is it Important/ of general interest? (This is highly subjective and where many of my papers strike out because it turns out they’re ahead of their time.  grrr.)

Update:  Here’s xykademiqz on the same topic for her science field.

Do you do a lot of referee reports?  How does your field handle them?

Null effects are fine, but you need to discuss power!

I like the way that a lot of social sciences are starting to push for publishing more studies that tried something plausible and then found nothing.  Null effects papers tend to be difficult to publish, which leads to publication bias, meaning you’re more likely to find something spurious than to not find something.

BUT.  One almost sure-fire way to find no effect is to have a sample size that is too small to pick up an effect.

If you find no effect, you need to discuss sample size.

If you find no effect from an experiment, then you really need to talk about the power analysis that you did *before* you ran the experiment that shows the sample size you would need to find an effect size.

And if you have a large magnitude that just isn’t significant, that isn’t as convincing as having an insignificant small magnitude or, even better, a small magnitude that flips sign depending on specification.

As the great Dierdre McCloskey says, statistical significance is not the same as oomph.  Or as I tell my students, meaningful significance is not the same as statistical significance.

A true null effect is something that has a small effect size, whether or not it is significant.  And if you find an insignificant null effect, then you have to discuss whether this is a true non-finding or if you just didn’t get enough observations.

Got that?  Null effect = fine, but it has to be a real null effect and not just a bad study.

Ask the grumpies: Can I really recommend accept with minor revisions in the first round?

Lucy asks:

I am trying to write a referee report on a paper and other than things I know are minutiae I have no comments!  Other than not showing anything causal (which the authors readily admit and isn’t really necessary for their question), I have no major criticism of anything they have done. The outlet is probably appropriate. So…do I seriously recommend publish as-is? Have you ever done that? It seems like such a cop out.

I think I have recommended some things be published with only minor revisions.  I know I’ve typed out under the Major section:  “I have no major concerns.”  And as an editor I’ve definitely gotten people making that recommendation, “Accept with only minor revisions”, even in the first round.  I just did one, in fact, that came back with “accept with minor revisions” from two reviewers in the first round.  And then I read it and was like, yeah, they should cut out that one section and see a copy-editor, but this is definitely an accept with minor revisions.

What you need to do so that the editor believes you (IMHO on the receiving end of these reports) is to explain in the cover letter why you think it doesn’t need revising.  So you say what you told me.  It exhaustively documents info, it doesn’t show anything causal but the authors are upfront about that and you don’t think it is necessary for them to show causation given the topic, the outlet is appropriate, etc.  The authors should be commended, etc.

It’s not enough to say, “accept with minor revisions” because then I’m all… should I trust you, or are you just lazy?  But if you can say why the paper is interesting and important/appropriate and anticipate problems that you don’t think are problems, then your letter is really helpful when I have to compare it to someone who, say, believes the paper should be rejected because it isn’t causal.  I had a situation like that once with two extremely enthusiastic reviewers and two who wanted to reject the paper outright and one of the rejects and one of the minor revisions were useless because they didn’t tell me anything useful.  If the second accept with minor revisions had told me why to accept, then the decision would have been a lot easier for me.  (Or if the other reject had said something other than, “this paper doesn’t cite [my papers]” even though it cited a literature review that contained said papers.)

Please use more topic sentences

In your technical writing.  Please!

What is a topic sentence, you ask?  Since they no longer seem to cover that in third grade…

A topic sentence is the first sentence in a paragraph that provides the main idea of the paragraph.  Essentially it introduces a paragraph and summarizes what the paragraph is going to say.  It isn’t, “Now we turn to Table 2”.   It isn’t, “[Author (DATE)] studies X.”  What does Table 2 say?  Why is it there?  Why are you talking about Author (DATE)?  What is the relationship to your paper?  Convey this information in the first sentence of each paragraph.

The topic sentence should tell you why that paragraph is there.  If you don’t know why that paragraph is there, then maybe it shouldn’t be.

This PSA brought to you by a grumpy rumbler who has had to do waaay too many referee reports recently.