Null effects are fine, but you need to discuss power!

I like the way that a lot of social sciences are starting to push for publishing more studies that tried something plausible and then found nothing.  Null effects papers tend to be difficult to publish, which leads to publication bias, meaning you’re more likely to find something spurious than to not find something.

BUT.  One almost sure-fire way to find no effect is to have a sample size that is too small to pick up an effect.

If you find no effect, you need to discuss sample size.

If you find no effect from an experiment, then you really need to talk about the power analysis that you did *before* you ran the experiment that shows the sample size you would need to find an effect size.

And if you have a large magnitude that just isn’t significant, that isn’t as convincing as having an insignificant small magnitude or, even better, a small magnitude that flips sign depending on specification.

As the great Dierdre McCloskey says, statistical significance is not the same as oomph.  Or as I tell my students, meaningful significance is not the same as statistical significance.

A true null effect is something that has a small effect size, whether or not it is significant.  And if you find an insignificant null effect, then you have to discuss whether this is a true non-finding or if you just didn’t get enough observations.

Got that?  Null effect = fine, but it has to be a real null effect and not just a bad study.


Ask the grumpies: Can I really recommend accept with minor revisions in the first round?

Lucy asks:

I am trying to write a referee report on a paper and other than things I know are minutiae I have no comments!  Other than not showing anything causal (which the authors readily admit and isn’t really necessary for their question), I have no major criticism of anything they have done. The outlet is probably appropriate. So…do I seriously recommend publish as-is? Have you ever done that? It seems like such a cop out.

I think I have recommended some things be published with only minor revisions.  I know I’ve typed out under the Major section:  “I have no major concerns.”  And as an editor I’ve definitely gotten people making that recommendation, “Accept with only minor revisions”, even in the first round.  I just did one, in fact, that came back with “accept with minor revisions” from two reviewers in the first round.  And then I read it and was like, yeah, they should cut out that one section and see a copy-editor, but this is definitely an accept with minor revisions.

What you need to do so that the editor believes you (IMHO on the receiving end of these reports) is to explain in the cover letter why you think it doesn’t need revising.  So you say what you told me.  It exhaustively documents info, it doesn’t show anything causal but the authors are upfront about that and you don’t think it is necessary for them to show causation given the topic, the outlet is appropriate, etc.  The authors should be commended, etc.

It’s not enough to say, “accept with minor revisions” because then I’m all… should I trust you, or are you just lazy?  But if you can say why the paper is interesting and important/appropriate and anticipate problems that you don’t think are problems, then your letter is really helpful when I have to compare it to someone who, say, believes the paper should be rejected because it isn’t causal.  I had a situation like that once with two extremely enthusiastic reviewers and two who wanted to reject the paper outright and one of the rejects and one of the minor revisions were useless because they didn’t tell me anything useful.  If the second accept with minor revisions had told me why to accept, then the decision would have been a lot easier for me.  (Or if the other reject had said something other than, “this paper doesn’t cite [my papers]” even though it cited a literature review that contained said papers.)

Please use more topic sentences

In your technical writing.  Please!

What is a topic sentence, you ask?  Since they no longer seem to cover that in third grade…

A topic sentence is the first sentence in a paragraph that provides the main idea of the paragraph.  Essentially it introduces a paragraph and summarizes what the paragraph is going to say.  It isn’t, “Now we turn to Table 2”.   It isn’t, “[Author (DATE)] studies X.”  What does Table 2 say?  Why is it there?  Why are you talking about Author (DATE)?  What is the relationship to your paper?  Convey this information in the first sentence of each paragraph.

The topic sentence should tell you why that paragraph is there.  If you don’t know why that paragraph is there, then maybe it shouldn’t be.

This PSA brought to you by a grumpy rumbler who has had to do waaay too many referee reports recently.