How to write a referee report

Seriously cribbed from A Guide for the Young Economist.  This is how the majority (though not all) of economists do it, and when I’ve reviewed for other fields I’ve been complimented on the organization, so I don’t think you can go wrong using this format even if you’re not an economist.

The letter to the authors

Start with a paragraph called “Summary”.  There’s some disagreement if these are still needed or just waste time, but I think if you do the summary paragraph right, it can be useful to both the editor and the authors.  The summary should NOT just be a restatement of the abstract.  It should be a summary of what the actual paper is about, and not what the authors think it’s about.  So, for example, if it’s an experiment, you would have a sentence saying what you think the authors are trying to do (Ex. The authors explore the effect of salt water vs. fresh water on underwater basket weaving.)  Then you say briefly what they actually did.  (The authors did a randomized controlled experiment using a student population in which…)  You might end with a statement about how they extrapolate their claims to a broader issue.  Often the abstract doesn’t actually fit what the paper is about– it makes much larger claims about what happened.  The summary should be neutral and describe what the authors actually did.  This is helpful to the editor to know what the paper is actually about before they read through it, and helpful to the authors in case what your understanding of the paper is about is different than what they intended.  They can fix their writing to make the paper more clear.  I find it helpful to focus on the method section and tables only for this part.

Then create three sections:

Major:

Minor:

Minutia:

Major should include things that you think must be fixed before the paper is published.  If the paper is a reject, then this is where reasons for rejection would go.  If the paper is an R&R, these are the items that must be addressed for sure.

Minor:  This is where smaller questions go.  You might have things that need clarification, things that are incorrect, additional robustness checks that are not make-or-break but should be addressed, and so on.  It is helpful to include page numbers with these.

Minutia:  This is where all the typos, page proofing, etc. stuff should go.  You’re recognizing that they’re small mistakes that the authors will want to fix, but that they’re not big deals.  These definitely need page numbers.  (If one of your recommendations is “spellcheck” because there are multiple spelling problems, I would put that under minor, as opposed to saying “should it be here instead of hear on the first sentence of page 28?” which would be a minutia, but YMMV.)

Always be polite in your referee report, even if the paper is ridiculous.  Do not make a reject/R&R recommendation within the paper.  (Also:  as an editor I can say for certain that positive letters don’t always lead to R&R recommendations and negative letters don’t always mean the person recommended reject.  It’s insane how some people can say different things depending on the audience.)

Advice is generally that you do not have to spend as much time on reject papers as you do on R&R– some people will say just stating the major points is enough if you plan to reject.  As a reviewer I generally try to give advice for making the paper better should it go to another journal or should the editor disagree with my assessment, but sometimes a paper is just not publishable so it doesn’t matter if they never fix the typo in footnote 17 even if I found it.  I’ve found editing at a lower tier journal that reviewers tend to over recommend revise and resubmit (they’ll be like, “the paper says that correlation is causation, but if they could only get at causation, this would be a great paper, R&R”), and the explanations people give me are much more important than their actual recommendations.  My colleague who edits at a top journal says reviewers over reject (“this is the best paper I’ve ever read, Reject”), so the explanations are important.  When I was editing a top field journal, reviewers tended to get it “right” on average.

The letter to the editor

You will also generally have a letter to the editor.  I find the best editors letters provide a concise summary of the letter to the authors and possibly elaborate on the context of your comments– Basically reiterate the major points that led to your decision of reject, or explain what must be fixed before publishing.  If you don’t have much to say because it’s an obvious accept, use this space to fight for the paper.  You don’t have to be anonymous in the letter to the editor, so you can say more things that put it into context or explain what you’re not sure about because it’s not your area of expertise or what you are sure about because you are an expert.  If you’re not sure if it should be R&R or Reject, here is a good place to say so and explain why– what are the pros and cons?  These pros and cons should also be in your letter to the authors, but you can provide more context in your letter to the editor.  You can also put disclaimers in the letter to the editor like, “I didn’t realize when I accepted this paper that it was written by a former coauthor” or “I reviewed this paper for a top journal earlier and recommended it be sent to this journal instead.” Some dudes who read this blog think that there should never be anything said to the editor that isn’t in the letter to the authors, but I strongly disagree.  I appreciate the reiteration of the major points of the review (especially since some people don’t use must be fixed as their delineation between major and minor sections, but instead use difficulty of fixing etc.) and any context that I should know about (and I really don’t need to know about that typo on footnote 17 unless the paper is a revise and resubmit, but not everybody keeps those things to the minutia section).

Special topic:  Top journals

For top journals (for which I have not yet been an editor but have done a lot of reviewing), you may want to keep in mind the following points:

1. Is it clean/well-done?  (This is the bare minimum)
2. Is it Novel? (Doesn’t always have to be, but it helps a lot… though you can’t be too novel or it gets rejected because it’s “not economics” even if it actually is.  grrrr.)
3. Does it make a major theoretical and/or empirical contribution to the field?  (Sometimes papers don’t need to add to empirics, but they do need to have a theory base even if not literally a theoretical model.)
4. Is it Important/ of general interest? (This is highly subjective and where many of my papers strike out because it turns out they’re ahead of their time.  grrr.)

Update:  Here’s xykademiqz on the same topic for her science field.

Do you do a lot of referee reports?  How does your field handle them?

Ask the grumpies: Should economists not teach anything about race?

SLAC prof asks:

In a tweet, Trevon Logan says

Message to 99.7% of Econ professors: This coming academic year, do NOT attempt to teach about race, racism, or racial inequality. Just don’t. Even if you’ve published papers on it. Respect this as a body of knowledge and refer students to appropriate academic units.

— Trevon D Logan (@TrevonDLogan) June 21, 2020

The whole thread has more information.  It makes me want to give up.  He says economists do race all wrong.  What do you think?  And what does one need to do/know to be qualified to teach about race?

Ok, so first off:  I am not black.  Also I know and hugely respect Trevon Logan and his work (and I’m fairly sure we referee each other’s papers and I’ve always been impressed with his!)

But I disagree with him.  I think this is ok for two main reasons:

First, I have had a relatively large number of black (mostly female) students, many of whom have taken some of these cross-campus classes he discusses, and they have always asked me for more on race, not less.  You just cannot teach health economics without discussing disparities (and many of the big papers in this area are from epidemiologists and demographers, not economists).  You cannot teach labor economics without having a huge section on discrimination, and while many of the white male economists working in this area have blinders on, it is fairly easy (if you have been listening to people, or if you’re female/minority) to point that out and modify their theories into something more realistic and less bigoted.  Like, of course taste-based discrimination exists, we don’t have competitive markets, duh.  (And current US events during my last semester’s class made it very clear that discrimination can lead to monopoly power, not just be a consequence of it.)  Theories of statistical discrimination should include incorrect stereotypes because we don’t have perfect information, honest to FSM.  Your (not privileged white male) students can generally point out these flaws themselves just using their own experiences and common sense.  You cannot teach public finance without talking about the political economy of race and how these programs affect different groups.  Heck, Political Economy is less than half a class without discussing race.  Similarly, Law and Economics (even if you’re planning on limiting to patents and contract law, race is still a factor!).  Sports economics!  You just cannot do justice to any subject that affects money or people without discussing how race impacts it.  So I include these topics and every year my students have more ideas for things to add.  (Like yes, in health economics we do need to talk about how white doctors have used black women’s bodies and DNA without their permission, you are absolutely right.  That would be a great addition to the Tuskeegee paper we already discuss.)

Second, I have listened to the troubles of our young black female faculty across campus (I was on a university-level thing to improve things, which we sort of did but also mostly didn’t … in any case, we did a lot of listening in addition to convincing the university to allow salary equity bumps and a few other things) who primarily teach these classes that Dr. Logan is suggesting we send our econ majors to.  It is really unfair to them to inundate them with mostly white male econ majors who have been taught that it’s just fine to play devil’s advocate and haven’t really examined their implicit biases at all.  I have enough trouble breaking them in in my intro stats classes.  Can you imagine how disruptive they would be in a discussion based class with women and minorities from what they consider to be lesser majors?  That is going to have huge negative spillovers.

I have other reasons to disagree which may be less ok, and I would modify his advice some.  (Note that since I wrote this post– several other people in the comments of the twitter thread have made these or similar suggestions.)

First off, I agree with him 100% that most of the white dudes in econ who gatekeep and work on racial discrimination start from racist assumptions and for many of them, their main goal is to show how it is Black people’s fault (or women’s fault etc.) for not being more like White men.  It’s only recently that economics has started thinking that no, maybe Black people and women are rational, they’re just playing a different game.  This problem can easily be solved by just saying, “Don’t teach any papers on race by white men (or by Roland Fryer who may be black but has serious issues).”  You can even modify this advice to “Teach only papers on race by black scholars (except not Roland Fryer).”  There’s plenty of great work by black scholars and some by other minorities and women that don’t start with racist assumptions or trying to bend evidence to “prove” racist ideas.  There are even textbooks and summary articles that would be great for lower-level undergraduate classes (William Darity Jr. is a good author/editor to start with).

And there are a LOT of white economists who could themselves benefit from reading this work.  Maybe they should start with So you want to talk about race and/or White Fragility and following Black scholars on twitter.  Then they can move on to articles in academic journals.

In terms of whether or not economists think about discrimination incorrectly… some of them do, but I think we benefit from looking at how different social sciences deal with race and discrimination.  NONE of them give a complete picture.  The assumptions and questions asked are different.  We gain tremendously from thinking about these different viewpoints and different ways of modeling.  (I took Race and the Economy from an amazing Black woman and she incorporated overviews from other fields in the class.  It can be done.)  I could go into huge detail about this, but that would get too long… suffice to say that these different viewpoints complement each other; they are not substitutes.  An economist can learn a lot from how anthropology, sociology, psychology and other fields conceptualize discrimination and other questions involving race.  (Insert rant about irritating white male gate-keepers in labor economics here who think innovation and interdisciplinarity is incorrect.)

Maybe the better advice would be for economist professors themselves to take a few classes across campus, or at the very least, read a textbook from another field, before adding race to their classes.  They should also read up on how to make their classroom more inclusive so that students don’t feel scared to speak up when the professor screws up.

As for me, I have been including race in my classes since I started and I cannot imagine stopping now.  The more I teach, the more I listen to my students, and the more I learn from them, which helps students the next time I teach.  It is a learning process for everybody.  Did I have some cringeworthy moments when I first started, probably, but minority students have been gentle with me and each year I’ve learned more and gotten better and future students benefit from that.

Update:  The more I talk with my colleagues interested in adding a race unit in their classes, the more I’m convinced that my suggestion about only using papers written by minorities is the correct one.  I had no idea that people didn’t know Becker was a huge racist misogynist jerk(!)  I mean, I thought everybody knew that.  People knew it back when he was still young, like decades ago.  So no, DO NOT read Becker in the raw original.  Many of his theory structures are lovely, but read them with the sexist and racist assumptions removed by someone else; there are great minority scholars who have explained the baseline theories and added to them, so go with them.  (William Spriggs talks about some of the problems still inherent today.)

I swear, my colleagues are all going to give up and just end up covering Bertrand and Mullainathan, though I did convince one to try Quillian et al. (in PNAS) instead.  Look, it’s not that B&M isn’t a great paper, it is, but the really horrible overlying thing is that it got into the AER because everybody, including labor economists who should have known better, thought this was the first time a correspondence audit had been done, completely ignoring ALL of the correspondence audits done by Black scholars or non-Americans– I learned about them in my undergrad economics class on Race in the economy.  What I mean is, I’m fairly sure that racism is the reason those earlier audits by black people aren’t known at all.  Quillian and coauthors do a good job of collecting them and plotting their results over time.  (It should have been published in Science, but the racist editor overruled like 7 referees who all said it was must publish.)  Quillian is also white, but he’s a sociologist, so maybe he gets a pass?  Plus he’s very nice.  I’m not sure if there are any minorities in the “et al” portion.  (Plus the econometrics textbook we use has B&M as one of their datasets and students replicate all the ttests and regressions, so it’s not adding that much for our majors.)  Any time I explain this to a White labor economist they get really mad at me because B&M is somehow the first hardcore proof they’ve ever seen about racism against black people other than those small scale in-person audits from like the 70s that somehow Jim Heckman “disproved”  in the 1990s (spoiler:  he didn’t really).

Update 2:  Last night we talked to a number of students and alumni (mostly underrepresented minorities) and they said to be careful to make sure that the lesson is integrated into the curriculum, and to not just have it as a separate unit unconnected with the rest of the class.

Ask the grumpies: How to subvert the Tragedy of the Commons

Leah asks:

Is there any way to subvert the tragedy of the commons, or are we doomed to that fate? I seem to remember learning some examples way back when I took environmental economics but they all escape me . . .

I just happen to teach a class on this!

The first way we learn about is with government setting property rights and facilitating costless Coasian bargaining.  In the canonical example, there’s a river and a factory and a fisherperson.  The factory pollutes the river which kills some fish.  If the factory owner owns the river, then the fisher can pay it to pollute less.  If the fisher owns the river, then the factory owner can pay them to allow some pollution.  There’s problems with this solution when there’s not costless bargaining, when there’s multiple fishers (that can cause a holdout problem) or multiple factory owners (and they don’t know which ones are causing the pollution), but that’s the “preferred” government intervention when it works because it leads to the least amount of deadweight loss.

Fancier versions of this solution include things like the government setting a specific number of pollution credits and allowing firms to bargain over them.  That’s the idea behind Cap and Trade.

When the Coasian solution is difficult to implement, generally because of bargaining problems or informational aysmmetry, the government can step in a bigger way.

First:  The government can mandate that firms not be allowed to pollute more than a certain amount or fish more than a certain amount or hunt more than a certain amount.  Associations can also take the role of government in order to say, prevent over-fishing, though it’s often harder for a non governmental association to enforce these kinds of mandates.  Mandates are most enforceable when there’s jailtime associated (not just a shell company going bankrupt), though that tends to be unpopular.

Second:  The government can tax things like pollution or things that cause pollution.  Think gasoline taxes or hunting fees.

Third:  The government can subsidize companies to not pollute or to not fish etc.  This option tends to be the most popular with industry.

All of these methods have situations in which they work better or worse than the other solutions.  With nuclear waste, you want a mandate because even a little bit of waste is bad.  With air pollution you might want a tax or subsidy or cap and trade system.  The government can make money with taxes or by selling property rights in a Coasian situation.  Companies tend to lobby for subsidies which makes them more politically feasible.

So the short answer is:  yes, government can subvert the tragedy of the commons.  Market failure is why there is an economic role for government and the tragedy of the commons is one of the causes of a main source of market failure (negative spillovers).  But we need political will for it to work.

Replacing a misogynistic mnemonic

It is easy to get backwards when doing t-tests, especially when you’ve first started.  You have to remember that big t and small p-values mean to reject.  Of course, if you’re a little dyslexic (undiagnosed) like me, when you get confused, you can go back and re-figure out that you want small amounts in the tails and the t-slice to be far away from the mean making it large etc., but that is really time consuming, especially if you’re in the middle of a lengthy problem set or a timed exam.

Many years ago, one of my students shared a dirty misogynistic mnemonic.  When p is low, she said, reject the ho.  This is clever and funny because p is both probability and slang for penis.  Ho is both how a null hypothesis is written and a derogatory term for female prostitute.  When one’s penis is at low mast, it makes sense that said penis-holder might not be purchasing the services of a prostitute.  And it rhymes.

But it’s both dirty (I got into trouble for saying “prick” meaning “jerk” early in my classroom career) and one should not be using derogatory terms for prostitutes or women anywhere, much less in an institution of higher learning.

More recently, I was explaining my conundrum in office hours and one of my sunshiny students came up with a much better mnemonic.  It isn’t quite as clever, but it’s just as memorable and it still rhymes.  “When p is lo, reject H-O!”  Like a cheerleading chant (aich – oh).

It makes me much happier.

What are some good non-racist, non-misogynist, non-ablist, non-patriarchy mnemonics that you know?

Ask the grumpies: whether or not to purchase insurance

H.I.P. Person (Home Insurance Purchasing Person) asks:

How do you decide if you need insurance for something? We are updating our home owners coverage and they are peddling the following:

1) service line coverage –  up to $10,000 per event
2) systems coverage (a/c, hvac, water heater, furnace, and the like) – up to $50,000 per problem
3) sewer back up (?? not sure of max coverage)

We live < 75 miles from the coast; <20 miles from two major water coastal inlets but  not in a flood zone. What hurricane related insurance should we own? The policy comes with wind/water but not flood and they don’t really want to sell us flood (they are offering #3 instead).

Are any of these worth it?  What price point would make them worth it or not?

DISCLAIMER:  We are not financial advisors.  Get advice from real professionals or do your own research before making important monetary decisions.

So in economics, you purchase insurance when the expected utility of the insurance is greater than the cost of the insurance.  So, assuming you knew how your utility function was shaped (the important part for this purpose is that you know your coefficient of risk aversion, in this case specifically how much you hate the possibility of loss), and you know the probability of a bad thing happening, you just multiply that probability by your expected utility plugging in the amount you’d be out if the bad thing happens and then add the probability that the bad thing doesn’t happen and multiply that by your expected utility plugging in the amount you’d have if the bad thing didn’t happen.  Problem is… in reality, we usually don’t know the probabilities of these negative events occurring (or even what those negative evens could be!) and we definitely have no clue about what our utility functions look like.

So how does one decide what to get insurance on in reality?  Well, if you have rough ideas of probabilities, you can look at the expected value of something happening.  Expected value is like expected utility, but it tells you what the break-even point is assuming you have no emotions.  You’re neither a gambler nor risk averse.  But risk averse you can look at those numbers and think to yourself, “How do I feel about this calculation?”  In general, the insurance company is out to make money, so they’ll be charging more than the expected value of the thing happening, but does the amount more that they’re charging seem reasonable to you?

If you don’t know the probabilities of something bad happening, you can still play with worst case scenarios:  If the bad thing happens, what would you have to pay out of pocket to fix it without insurance?  What would you have to pay out of pocket with their insurance?  Is the peace of mind for the difference between those two numbers worth what they’re charging?  (And again, if you have rough ideas of how probable these events are, you can factor that in as well).

Another thing to do is to google around, preferably with reputable sites, to see what kinds of insurance are usually a good idea and what kinds are generally scams.  You can ask people around you too, though people often do things that don’t make sense if there’s good marketing on the part of the insurance company or if they’re more credit constrained or risk seeking than you are.

An important thing to note is that you want insurance to insure against risk.  You don’t want it as a pre-payment for things you’re going to pay for eventually.  You don’t want a high monthly cost if you can avoid it by sharing some of the pain should disaster strike through a higher deductible.  The goal isn’t to save money, it’s to smooth your consumption over good and bad states of the world.  Don’t try to beat insurance– in the best states of the world you give them money and they never give you money.  But you want it there when disaster strikes if you can’t handle the disaster on your own.

If it were me, I’d definitely eschew the service line coverage unless it was really cheap– we can afford a 10K emergency.  Since the limit is capped, it is not actually useful insurance unless paying up to 10K would be devastating.  (Capped insurance is often a red flag– if they stop paying after a certain amount is spent you may be better off self-insuring because if something really terrible happens they’re not going to be much use, and self-insuring means you’re not paying for the additional administrative costs of going through them.)

Systems coverage again, we probably wouldn’t pay… replacing a/c, hvac, water heater, furnace etc. is all stuff that has to get done some time anyway (and none of these should cost 50K, unless that’s including the damage after a water heater explodes or something) so paying them is like pre-paying for bills you’re going to have, but it is likely you’re going to have less choice about how to make those replacements and you’ll be paying their administrative costs over what you’d be paying if you did these things yourself.  And, again, it’s capped.  This is unlikely to be good insurance.

Sewer back-up is the only one of these that doesn’t sound 100% scammy.  Check all your other insurances to see if this is covered under them.  Estimate how much a sewer backup could destroy.  Make sure that it’s unlimited covered and not capped and there are no other strings attached.  Think about the probability of this happening.  Look at how much they’re charging for it.  Then go with your gut.

We have no idea about hurricane insurance.  The internet has a bunch of pages about it, noting you should get wind and flood on top of home, but I’m not getting a good idea of what numbers you should be looking at or even how to make that calculation.  You may also want to look into the different deductibles they offer, because none of these sound particularly cheap, but it may be that if you have a high deductible the insurance cost will be more reasonable (assuming you can afford the deductibles).

Good luck!

Grumpy Nation, do you have any better advice for H.I.P.?

 

Ask the grumpies: Privatizing nation’s air controllers?

Crone asks:

opinion on privatizing our nation’s air controllers. I oppose but was told the whole system should be moved to computer based GPS system and then Highways in the Sky for planes could be free form making flights faster and private industry can do this more rapidly than government. (I was in social situation so could not say I have never known a single computer system that did not ‘go down’ or ‘have ‘undocumented features ‘ so how would that work…) The topic of pipelines that ‘will not fail but ALL LEAK at some time’ had already come up.~~ I had been assured I was wrong on that point and ignorantly female. SO, back to air controllers: If this would be profitable for private companies to do why isn’t it done profitably or better by public government?

OMG, the WORST IDEA. OMG OMG OMG.

Well, it’s only the WORST IDEA if you think that airplane passengers are more important than prisoners. If you think prisoners are people too and should have rights, then privatizing prisons is actually the worst idea and this is only second worst. I guess there’s also privatizing foster care systems… if you think all people are equal then that might be slightly above air traffic control but still below prisons in potential harm done by privatization. (Foster care systems empirically aren’t as bad as prison systems, even though the potential is there to be as bad. This has to do with better state oversight.)

I had a section on privatizing public systems in one of my classes last semester and students brought in stuff– if I’d known it would come up as an ask the grumpies I’d have taken a picture of the whiteboard commonalities of when it works and doesn’t that we came up with. It can be ok, but it depends on a lot of stuff and it really shouldn’t be something where you know, people could die.

Ugh, so no, not you being ignorantly female. There’s a reason there’s a role for government for various systems.

We generally think that there is a potential role for government intervention when there is market failure in the competitive markets.  One form of market failure comes from monopolies.  Something like air-traffic control is what we call a “natural monopoly”– natural monopolies occur when it just doesn’t make sense for more than one company to be in one market.  A lot of utilities are in this kind of situation– where it doesn’t make sense for two companies to lay down pipes or what-have-you.  (You can also have government-private partnerships, where, for example, the government owns the rail-lines but allows different companies to pay to use them.)  Air traffic control is an example of a natural monopoly.  At an airport, it makes sense for only one company to do the air-traffic control.  Any more could lead to planes, for example, hitting each other.

The government in this situation could still allow private contractors to bid on the ability to be that one company doing all the air traffic controlling.  Unfortunately, air traffic control benefits a lot from experience and there are switching costs when an old company leaves and a new one takes its place.  Those switching costs could lead to not just inefficiency but also death.   Finally, oversight is really important with privatization.  Unlike the government, companies can just go out of business when they cut costs so much that people die, so they don’t have as much of an incentive to stay safe when it means cutting into profits.  Government can combat that by making it costly for them to cut corners before someone ends up dead, but that oversight comes at a cost.  Those costs could be large enough (and the possibility of bribes could be high enough) that it makes sense for the government just to do it itself.

 

What does statistical significance mean?

One of my students sent me this article because we spend some time in class covering Type 1 and Type 2 errors.

All the .05 threshold means is that you have a false positive 1/20 times.  A .005 threshold would say you’re getting a false positive 1/200 times.  So by moving to a .005 threshold, you’re less likely to get a false positive.  That’s good, right?  In common parlance, we’d be less likely to send an innocent person to jail.

Well, that depends.  At the .005 threshold, we’re more likely to get a false negative than you would at the .05 level.  That means we’d be more likely to get a guilty person go free.  (Indeed, the only guaranteed way to send no innocent people to jail would be to send nobody to jail.  I, for one, am happy that folks like Charles Manson are behind bars.)

It isn’t as easy as saying, oh we should just switch to .005.  When you adjust the p-value you’re making trade-offs between type 1 and type 2 error.  With a lower p-value threshold you’re going to be getting a lot more false negatives even with fewer false positives.  What we always need to be cognizant of when we’re doing policy is that significance isn’t everything– we also have to think about what the damage is if this information turns out to be incorrect.  For example, doctors recommend that pregnant women should heat up cold cuts if they’re worried about listeria, which is a very low probability event but if it happens it’s horrible.  It’s pretty easy to avoid room temperature cold-cuts for 9 months, so unless there’s some other difficulties attached to diet, women will probably follow this recommendation.  (And if one accidentally eats room temperature coldcuts while pregnant, one shouldn’t freak out because the probability of getting listeria is very low!)  But if we’re talking about something like doing chemotherapy or surgery, that’s a much more onerous action and we might want to be more sure we need it before going ahead with it.

Another thing to note is that the article talks about how physics and genetics have already made this switch, while most social sciences haven’t.  One big difference between the fields that have made the switch and the fields that have not is how easy it is to get large samples.  A larger sample size will make it so your sample behaves more and more like the population that you’re trying to study.  We can reduce both Type 1 and Type 2 errors simply by increasing the sample size.  So why don’t we do that?  Well, it turns out that increasing the sample size can be very very expensive when you’re dealing with people and behavior.  Sometimes doing the study with a large enough sample to get 80% power and an alpha of .005 might be more expensive than just throwing that same money at the intervention you’re trying to decide about, whether or not it actually works.  There probably is some resistance because people in these fields want to be able to publish their 5% results, but that’s not the main or only reason we haven’t yet made the switch.  Research is complicated and expensive and we have to make trade-offs.

The context for these really does matter, and you shouldn’t necessarily put off making policy choices just because your sample size is too small to get significance (or to make policy changes just because you have significance).  You always have to be aware of the costs and the benefits.

 

(Incidentally, in case he comes across this, Hi Dan!  I’m assuming that the reporter greatly simplified your arguments here because I know you must know this stuff.)

Ask the Grumpies: Where to learn economics?

Leah asks:

I’d love to learn some basic econ. Where’s a good place to start that is not too arduous but is also accurate?

I know there are a lot of Econ for laypeople books out there (Freakanomics being the most famous), but when people ask me this question I always stick to two textbook recommendations.  The first is the Intermediate Microeconomics by Robert Frank called Microeconomics and Behavior.  The second is Public Finance by Jonathan Gruber.

I love Frank because he discusses microeconomics in a way that contrasts how the rational person would behave with how people actually behave.  This I think makes the theory more believable and more powerful.

Public Finance I really think ought to be taught in high school.  If you want to understand the role for government, it is a must read.  So much of what is going on with healthcare right now violates basic economic principles and after reading about adverse selection, you, too, will understand why.

Note for these that you do not at all need to buy the most recent edition.  The 1998 edition of Frank is fine for understanding the basics.  The first edition of Public Finance by Gruber is still a fantastic read.  Get whatever is available and cheap.

What economics tomes/videos do you recommend, Grumpy Nation?

We live in interesting times

We live in interesting times.

I often think that this is must have been like what it felt for our parents growing up in the 60s.  Marches and riots and violence in the news all the time, but a sense that progress was finally being made.

Life was so much easier during the booming 90s.  Of course, that’s not really true.  Life was easier for us white folk, and we just didn’t know about what was going on elsewhere.  The Rodney King riots were a glimpse into what life was like for others, but the rest of us really stopped paying attention until recently.

One of the reasons Hamilton is doing so well is that it isn’t really about the 18th century.  It is about today.  This musical number really encapsulates it.

Change comes slowly and then it comes all at once.

Change comes with violence.  Or rather, that’s what we perceive.  Those of us who are sheltered and privileged.  The violence was always there.  On the plantations.  Against share-croppers.  Burning crosses on lawns.  Killing people in our cities.  Lynching, rape, murder, beatings.  Those of us who are outside don’t notice.  We believed things were accidents and tragedies or isolated incidents or provoked by criminals.  But that’s not what was going on.  That’s not what is going on.

Like now, change happens when violence is made visible.  Then violence escalates.  Violence escalates because the people in power, the ones doing the terror attacks against minorities, the ones subjugating their wives, girlfriends, and daughters, are afraid.  And they are afraid.  And violence is their only real weapon.

Which isn’t actually true.  Violence is not and has never been their only real weapon.

The Voting Rights Act was in response to their hold on local governments.  They own state and local governments again.  We MUST organize locally.  We must pay attention to downstream races.  We must run candidates even in red areas.

They’ve owned the media before, they own some of the media now.  Fox News isn’t the first news organization to have a racist misogynist agenda.  Not the first media organization to sway angry poor uneducated white men for their own causes.  It makes sense for uneducated white losers to want to keep women and minorities down– if they don’t have them to scapegoat and feel superior to, then they’ll be at the bottom of whatever metaphor you can think of.  It doesn’t make as much sense for the people who control these empires.  Why are there evil rich people?  Is it because they want more power than their horrible rich white associates?  But isn’t it better to be a Philanthropist than a Bond Villain?

Revolution means progress.  But revolutions are rarely easy.  Those in power fight back to maintain the status quo.

It’s best when revolutions occur with the fewest lives lost.  With the least blood spilt.

I think there’s a politician and bureaucrat who can help the revolution shed less blood while moving forward.  But she can only do it if she gets support downstream.  Senators.  Representatives.  State Government.  VOTE.  WRITE.  Be angry.  Protest.  Support protestors.  Become woke and stay it, even when the media moves on to the next story.  We want a government for all people, not just some of the people.

And after this movement dies down, we’ll still have a long way to go.  But let’s go as far as we can towards equality of opportunity, freedom, peace, and happiness as we can, so that maybe it won’t be as hard or dangerous next time around.  And so people can live closer to their best lives while we wait for the next revolution to bring them closer still.

#Imwithher

How to write a power-point discussion (economics-specific)

The goal of a good discussion is to explain to the audience where the paper fits into the general social science/policy framework and to help the paper improve for the future.  The goal is not to destroy a paper but to improve it (see exception below).  Discussants are serving science!

1. Frame question— why is it important?  (You can mention your own work here if applicable.)
2. Briefly summarize paper.  If the presenter is great, you will be able to skip the summary or only go over what you see as the most important parts.  If the presenter is terrible, your audience will really appreciate figuring out what they just heard, so it’s good to be thorough on your slides if you don’t know a priori how good the presenter will be.  If applicable, here would be a great place to take the author’s work through a “sniff test”– Bridgette Madrian is one of the best discussants I’ve seen, and one of my favorite discussions of hers was where she took a person’s paper (on whether or not we need 70% of our income after retirement) and applied it to her own life with a spreadsheet and came to the conclusion that the paper’s thesis was plausible.  Sometimes discussants will call up experts in the industry to ask their qualitative opinion.  Really great discussants will sometimes replicate or extend with another dataset.  None of these things are necessary, but if they’re easy for you or an RA to do, they can really push you to be memorable (though being invited to discuss more papers is not necessarily something you want to do!).
3. Constructively point out problems with the paper and suggest solutions (if any).  Don’t be a dick.  Frame these as questions to think about, how big a problem you think they are etc . Don’t use this part as a place to talk about why your work is awesome and theirs sucks.  If you do mention your work in this spot, use it only as a place to commiserate with standard problems and suggest solutions that could work for them.
4. Extensions for the future, broader impact.  Here’s a place where you can talk up your own work if it is related and can speak to the paper you’re discussing.

How many slides do you want?  Fewer than the number of minutes you have to present.  It is better to go short than to go long.

Special cases:

1.  The authors haven’t actually done anything yet:  Spent the majority of your time on why this is an interesting question and suggestions for future work.  (Also ok to use a chunk of your time talking about your own related work.)  Use the word “preliminary” a lot.
2. The authors clearly haven’t addressed causality but causality needs to be addressed (or any other major elephant in the paper issue):  Spent the majority of your time on why this is an interesting question.  Talk about the problems of getting to causality and (if easy for you to do) what other authors have done and (if easy for you to do) the problems with what they’ve done (or if not problematic, then suggest these authors follow).  Gently mention that causality is something that these authors need to think about.  The audience will understand.  Then suggest future work (which will include really nailing down causality).
3. You don’t get the paper to discuss until the night before at 3am:  Feel free to spend the entire time talking about your own work, or to come up with something off the cuff while they’re giving the presentation (it is AOK to note that you did not get the paper until the night before, but that should be the extent of your dickishness).
4. The paper is poorly done and the results, if taken at face value, will do real harm to people, particularly those from marginalized groups:  In this case, it is ok to firmly and politely destroy the paper for shoddy craftsmanship.  You can do so in a professional manner in steps 2 and 3. You’re still not being a dick, but you don’t have to frame things as questions to think about but as real methodological problems.   It’s ok to throw around the terms “dangerous” and “needs stronger proof”.  It’s a shame that there are still guys (and the occasional woman) who write papers with sexist/racist agendas who ignore basic science in order to prove that wealthy white men are superior and deserve their privilege, but there are.  They shouldn’t be allowed to do bad science.

Academic readers– is this about right?  What things are the same or different in your discipline?  Any other tips?