Null effects are fine, but you need to discuss power!

I like the way that a lot of social sciences are starting to push for publishing more studies that tried something plausible and then found nothing.  Null effects papers tend to be difficult to publish, which leads to publication bias, meaning you’re more likely to find something spurious than to not find something.

BUT.  One almost sure-fire way to find no effect is to have a sample size that is too small to pick up an effect.

If you find no effect, you need to discuss sample size.

If you find no effect from an experiment, then you really need to talk about the power analysis that you did *before* you ran the experiment that shows the sample size you would need to find an effect size.

And if you have a large magnitude that just isn’t significant, that isn’t as convincing as having an insignificant small magnitude or, even better, a small magnitude that flips sign depending on specification.

As the great Dierdre McCloskey says, statistical significance is not the same as oomph.  Or as I tell my students, meaningful significance is not the same as statistical significance.

A true null effect is something that has a small effect size, whether or not it is significant.  And if you find an insignificant null effect, then you have to discuss whether this is a true non-finding or if you just didn’t get enough observations.

Got that?  Null effect = fine, but it has to be a real null effect and not just a bad study.

Ask the grumpies: Favorite class outside your major?

Leah asks:

What was your favorite class outside of your major and why?

#1  German, choral conducting, maybe that one English class where we read mystery novels.

#2  History, probably the British monarchs class because the prof for that one was especially awesome. (I would have been a history major, except the prof that I had for the required freshman seminar was a total a@#$@# and he taught a bunch of required classes, so screw that, and thus I ended up on a more potentially lucrative path.)

How about the rest of Grumpy Nation?

Ask the grumpies: Writing an external tenure letter

Tenured economist asks:

I was asked to write an external letter for a tenure case. Do you have any advice to share? We don’t use these in our tenure cases so I have never even seen an example! How long/detailed are they usually?

The following is based on external letters we’ve gotten in the tenure cases I’ve sat on so far.  We’d love to hear from the Grumpy Nation for people with more extensive experience and with experience in different fields.

There’s a lot of variation in these letters even from economists.

Usually they’re 1-3 pages long (single spaced with extra spaces between paragraphs, 12 point font, TNR, etc. give or take). Here’s what I’ve seen generally:

You don’t have to give a recommendation yes/no if you don’t want to. If you do, it can either be based on, “They would get tenure at [my university]” or “They should get tenure at their university”

You start by saying if you’re aware of the person’s work if you are aware of it, and if so whether or not you know the person personally and in what context. If you’re not aware of the person’s work you can choose to say that or to not say that.

Then you talk about the different strands of literature and put them in context for the committee. Talk about their quality and how they fit into the broader literature.

If there’s other items they ask you to address like teaching or service, then address those as well. We specifically ask for it to be focused on research and fit within the broader community (so potentially service to the profession if they have any) because we’re an R1.  SLACs, policy schools, and business schools might have different things they care about so if there’s something that the specific type of institution cares about you might address that.  Ex. teaching, media visibility, etc.  If there are potential things you might think would be concerning, like lack of single authored papers, you can talk about that as well and why that may or may not be a concern in this specific case.

That’s really about all there is to it.  The hard part is reading through the articles and figuring out their worthiness, especially you don’t have a helpful overview letter written by the applicant that puts it into perspective for you.

Ask the grumpies: Can I really recommend accept with minor revisions in the first round?

Lucy asks:

I am trying to write a referee report on a paper and other than things I know are minutiae I have no comments!  Other than not showing anything causal (which the authors readily admit and isn’t really necessary for their question), I have no major criticism of anything they have done. The outlet is probably appropriate. So…do I seriously recommend publish as-is? Have you ever done that? It seems like such a cop out.

I think I have recommended some things be published with only minor revisions.  I know I’ve typed out under the Major section:  “I have no major concerns.”  And as an editor I’ve definitely gotten people making that recommendation, “Accept with only minor revisions”, even in the first round.  I just did one, in fact, that came back with “accept with minor revisions” from two reviewers in the first round.  And then I read it and was like, yeah, they should cut out that one section and see a copy-editor, but this is definitely an accept with minor revisions.

What you need to do so that the editor believes you (IMHO on the receiving end of these reports) is to explain in the cover letter why you think it doesn’t need revising.  So you say what you told me.  It exhaustively documents info, it doesn’t show anything causal but the authors are upfront about that and you don’t think it is necessary for them to show causation given the topic, the outlet is appropriate, etc.  The authors should be commended, etc.

It’s not enough to say, “accept with minor revisions” because then I’m all… should I trust you, or are you just lazy?  But if you can say why the paper is interesting and important/appropriate and anticipate problems that you don’t think are problems, then your letter is really helpful when I have to compare it to someone who, say, believes the paper should be rejected because it isn’t causal.  I had a situation like that once with two extremely enthusiastic reviewers and two who wanted to reject the paper outright and one of the rejects and one of the minor revisions were useless because they didn’t tell me anything useful.  If the second accept with minor revisions had told me why to accept, then the decision would have been a lot easier for me.  (Or if the other reject had said something other than, “this paper doesn’t cite [my papers]” even though it cited a literature review that contained said papers.)

Please use more topic sentences

In your technical writing.  Please!

What is a topic sentence, you ask?  Since they no longer seem to cover that in third grade…

A topic sentence is the first sentence in a paragraph that provides the main idea of the paragraph.  Essentially it introduces a paragraph and summarizes what the paragraph is going to say.  It isn’t, “Now we turn to Table 2”.   It isn’t, “[Author (DATE)] studies X.”  What does Table 2 say?  Why is it there?  Why are you talking about Author (DATE)?  What is the relationship to your paper?  Convey this information in the first sentence of each paragraph.

The topic sentence should tell you why that paragraph is there.  If you don’t know why that paragraph is there, then maybe it shouldn’t be.

This PSA brought to you by a grumpy rumbler who has had to do waaay too many referee reports recently.

Stocks and bonds, Writing and outreach

I had an idea.  Follow me, here:

For academic careers, writing is like investing in stocks.  Outreach and translational research are like investing in bonds.

Stocks and writing:  Get lots while you’re young.  You need to write prolifically enough to get tenure, and gain the national or international reputation you need for those outside letters.  Spread your name, become known in your field.  Start early.  Because the return is uncertain, put a lot of writing out there in the world (and buy stocks).  Stocks are a good investment when you have a long timeline until retirement; you have time to weather the ups-and-downs of the market and can have a higher tolerance for risk, in exchange for possibly higher returns.

Bonds and outreach/translation:  These are more effective when you’re older.  When you’re more experienced in your field, you have more experience and a reputation that you can leverage for influence.  Research-wise, you’ve got a better idea of what works and what’s worth developing further, as well as potential pitfalls and objections.  You also know people who can help spread your ideas.  You may have more time to devote to making the world a better place.  When you’re closer to retirement, you also want the safety and security of bonds: potentially lower return, but steady.

In financial investing, as in an academic career, you’ll need a balance and variety throughout your life.  You might want to be doing both of these things (and more!) at all times, but in varying ratios.  Diversify and rebalance your portfolio and life.

This idea: off the wall, or right on target?  Tell me, Grumpeteers.

Where do I get my research ideas?

This year I have been giving an awful lot of talks.  Along with these talks, I’ve been meeting with a lot of graduate grad students during my visit.  A  common question I get when I meet with a group of students (you know, the ones with free time) is how I get my research ideas.  This usually comes from students who are floundering without a dissertation topic.  I thought I’d write up my answer.

  1. First, I get ideas from my contrarian nature.  Perhaps it’s my math training, but I am always looking for a counter-example, I am always questioning statements taken to be true.  My own job market paper topic, in fact, was a reaction from a statement one of my professors had made in a second-year class that struck me as possibly not true and when I looked into it, I found very little research on the topic.  I figured out how to test it better than the one or two previous papers, and voila, an amazing paper.
  2. Another place to get ideas that haven’t been worked on over and over is to think about your own unique experiences.  This can be something as broad as thinking about your own female perspective on sexist things that your male-dominated field takes for granted (ex. all the new research coming out showing that women aren’t irrational, they’re just working under different constraints) or as specific as a public program that not many people know about but you know lots about because your grandfather was on it.  You have lots of unique things that you bring to your discipline.  Think about what they are.  Think about who you know.  Look at the broader world around you and question it.
  3. It is ok to start out feeling like you keep coming up with ideas that have already been done– when I started out, it seemed like when I started the lit review I’d find that the exact paper I wanted to write had been written 10 years ago.  But my next idea had been published 3 years ago.  And the one after that, maybe just out.  Eventually I started coming up with ideas that were working papers.  And then new papers.  You may also find yourself in the situation where you’re half done with a paper and it seems like you’ve just been scooped– but you haven’t been really– it is unlikely your paper is identical to this other one and if it is, you can still change things, pursue different directions, answer some things better, etc. to differentiate it.  You want to be working in a hot field because it means your question is important.  See if you can create conference panels with this other paper.
  4. It gets a lot easier once you’ve gotten immersed.  After you’ve started a project, you start realizing there are huge gaps in the literature– things you really need to know now in order to fully answer your question but that are themselves their own projects.  You’ll also come up with new questions that your project has provided you… if this is true, then why this other thing?
  5. If you don’t do a perfect job, that means future people will fill in the gaps in your literature later!  It’s kind of exciting seeing people do a better job than you did because they are taking your paper as a starting point.  You know, so long as they cite you.

Where do you get your ideas?  What advice would you give current graduate students looking for inspiration?