What does statistical significance mean?

One of my students sent me this article because we spend some time in class covering Type 1 and Type 2 errors.

All the .05 threshold means is that you have a false positive 1/20 times.  A .005 threshold would say you’re getting a false positive 1/200 times.  So by moving to a .005 threshold, you’re less likely to get a false positive.  That’s good, right?  In common parlance, we’d be less likely to send an innocent person to jail.

Well, that depends.  At the .005 threshold, we’re more likely to get a false negative than you would at the .05 level.  That means we’d be more likely to get a guilty person go free.  (Indeed, the only guaranteed way to send no innocent people to jail would be to send nobody to jail.  I, for one, am happy that folks like Charles Manson are behind bars.)

It isn’t as easy as saying, oh we should just switch to .005.  When you adjust the p-value you’re making trade-offs between type 1 and type 2 error.  With a lower p-value threshold you’re going to be getting a lot more false negatives even with fewer false positives.  What we always need to be cognizant of when we’re doing policy is that significance isn’t everything– we also have to think about what the damage is if this information turns out to be incorrect.  For example, doctors recommend that pregnant women should heat up cold cuts if they’re worried about listeria, which is a very low probability event but if it happens it’s horrible.  It’s pretty easy to avoid room temperature cold-cuts for 9 months, so unless there’s some other difficulties attached to diet, women will probably follow this recommendation.  (And if one accidentally eats room temperature coldcuts while pregnant, one shouldn’t freak out because the probability of getting listeria is very low!)  But if we’re talking about something like doing chemotherapy or surgery, that’s a much more onerous action and we might want to be more sure we need it before going ahead with it.

Another thing to note is that the article talks about how physics and genetics have already made this switch, while most social sciences haven’t.  One big difference between the fields that have made the switch and the fields that have not is how easy it is to get large samples.  A larger sample size will make it so your sample behaves more and more like the population that you’re trying to study.  We can reduce both Type 1 and Type 2 errors simply by increasing the sample size.  So why don’t we do that?  Well, it turns out that increasing the sample size can be very very expensive when you’re dealing with people and behavior.  Sometimes doing the study with a large enough sample to get 80% power and an alpha of .005 might be more expensive than just throwing that same money at the intervention you’re trying to decide about, whether or not it actually works.  There probably is some resistance because people in these fields want to be able to publish their 5% results, but that’s not the main or only reason we haven’t yet made the switch.  Research is complicated and expensive and we have to make trade-offs.

The context for these really does matter, and you shouldn’t necessarily put off making policy choices just because your sample size is too small to get significance (or to make policy changes just because you have significance).  You always have to be aware of the costs and the benefits.

 

(Incidentally, in case he comes across this, Hi Dan!  I’m assuming that the reporter greatly simplified your arguments here because I know you must know this stuff.)

Advertisements

Ask the Grumpies: Where to learn economics?

Leah asks:

I’d love to learn some basic econ. Where’s a good place to start that is not too arduous but is also accurate?

I know there are a lot of Econ for laypeople books out there (Freakanomics being the most famous), but when people ask me this question I always stick to two textbook recommendations.  The first is the Intermediate Microeconomics by Robert Frank called Microeconomics and Behavior.  The second is Public Finance by Jonathan Gruber.

I love Frank because he discusses microeconomics in a way that contrasts how the rational person would behave with how people actually behave.  This I think makes the theory more believable and more powerful.

Public Finance I really think ought to be taught in high school.  If you want to understand the role for government, it is a must read.  So much of what is going on with healthcare right now violates basic economic principles and after reading about adverse selection, you, too, will understand why.

Note for these that you do not at all need to buy the most recent edition.  The 1998 edition of Frank is fine for understanding the basics.  The first edition of Public Finance by Gruber is still a fantastic read.  Get whatever is available and cheap.

What economics tomes/videos do you recommend, Grumpy Nation?

We live in interesting times

We live in interesting times.

I often think that this is must have been like what it felt for our parents growing up in the 60s.  Marches and riots and violence in the news all the time, but a sense that progress was finally being made.

Life was so much easier during the booming 90s.  Of course, that’s not really true.  Life was easier for us white folk, and we just didn’t know about what was going on elsewhere.  The Rodney King riots were a glimpse into what life was like for others, but the rest of us really stopped paying attention until recently.

One of the reasons Hamilton is doing so well is that it isn’t really about the 18th century.  It is about today.  This musical number really encapsulates it.

Change comes slowly and then it comes all at once.

Change comes with violence.  Or rather, that’s what we perceive.  Those of us who are sheltered and privileged.  The violence was always there.  On the plantations.  Against share-croppers.  Burning crosses on lawns.  Killing people in our cities.  Lynching, rape, murder, beatings.  Those of us who are outside don’t notice.  We believed things were accidents and tragedies or isolated incidents or provoked by criminals.  But that’s not what was going on.  That’s not what is going on.

Like now, change happens when violence is made visible.  Then violence escalates.  Violence escalates because the people in power, the ones doing the terror attacks against minorities, the ones subjugating their wives, girlfriends, and daughters, are afraid.  And they are afraid.  And violence is their only real weapon.

Which isn’t actually true.  Violence is not and has never been their only real weapon.

The Voting Rights Act was in response to their hold on local governments.  They own state and local governments again.  We MUST organize locally.  We must pay attention to downstream races.  We must run candidates even in red areas.

They’ve owned the media before, they own some of the media now.  Fox News isn’t the first news organization to have a racist misogynist agenda.  Not the first media organization to sway angry poor uneducated white men for their own causes.  It makes sense for uneducated white losers to want to keep women and minorities down– if they don’t have them to scapegoat and feel superior to, then they’ll be at the bottom of whatever metaphor you can think of.  It doesn’t make as much sense for the people who control these empires.  Why are there evil rich people?  Is it because they want more power than their horrible rich white associates?  But isn’t it better to be a Philanthropist than a Bond Villain?

Revolution means progress.  But revolutions are rarely easy.  Those in power fight back to maintain the status quo.

It’s best when revolutions occur with the fewest lives lost.  With the least blood spilt.

I think there’s a politician and bureaucrat who can help the revolution shed less blood while moving forward.  But she can only do it if she gets support downstream.  Senators.  Representatives.  State Government.  VOTE.  WRITE.  Be angry.  Protest.  Support protestors.  Become woke and stay it, even when the media moves on to the next story.  We want a government for all people, not just some of the people.

And after this movement dies down, we’ll still have a long way to go.  But let’s go as far as we can towards equality of opportunity, freedom, peace, and happiness as we can, so that maybe it won’t be as hard or dangerous next time around.  And so people can live closer to their best lives while we wait for the next revolution to bring them closer still.

#Imwithher

How to write a power-point discussion (economics-specific)

The goal of a good discussion is to explain to the audience where the paper fits into the general social science/policy framework and to help the paper improve for the future.  The goal is not to destroy a paper but to improve it (see exception below).  Discussants are serving science!

1. Frame question— why is it important?  (You can mention your own work here if applicable.)
2. Briefly summarize paper.  If the presenter is great, you will be able to skip the summary or only go over what you see as the most important parts.  If the presenter is terrible, your audience will really appreciate figuring out what they just heard, so it’s good to be thorough on your slides if you don’t know a priori how good the presenter will be.  If applicable, here would be a great place to take the author’s work through a “sniff test”– Bridgette Madrian is one of the best discussants I’ve seen, and one of my favorite discussions of hers was where she took a person’s paper (on whether or not we need 70% of our income after retirement) and applied it to her own life with a spreadsheet and came to the conclusion that the paper’s thesis was plausible.  Sometimes discussants will call up experts in the industry to ask their qualitative opinion.  Really great discussants will sometimes replicate or extend with another dataset.  None of these things are necessary, but if they’re easy for you or an RA to do, they can really push you to be memorable (though being invited to discuss more papers is not necessarily something you want to do!).
3. Constructively point out problems with the paper and suggest solutions (if any).  Don’t be a dick.  Frame these as questions to think about, how big a problem you think they are etc . Don’t use this part as a place to talk about why your work is awesome and theirs sucks.  If you do mention your work in this spot, use it only as a place to commiserate with standard problems and suggest solutions that could work for them.
4. Extensions for the future, broader impact.  Here’s a place where you can talk up your own work if it is related and can speak to the paper you’re discussing.

How many slides do you want?  Fewer than the number of minutes you have to present.  It is better to go short than to go long.

Special cases:

1.  The authors haven’t actually done anything yet:  Spent the majority of your time on why this is an interesting question and suggestions for future work.  (Also ok to use a chunk of your time talking about your own related work.)  Use the word “preliminary” a lot.
2. The authors clearly haven’t addressed causality but causality needs to be addressed (or any other major elephant in the paper issue):  Spent the majority of your time on why this is an interesting question.  Talk about the problems of getting to causality and (if easy for you to do) what other authors have done and (if easy for you to do) the problems with what they’ve done (or if not problematic, then suggest these authors follow).  Gently mention that causality is something that these authors need to think about.  The audience will understand.  Then suggest future work (which will include really nailing down causality).
3. You don’t get the paper to discuss until the night before at 3am:  Feel free to spend the entire time talking about your own work, or to come up with something off the cuff while they’re giving the presentation (it is AOK to note that you did not get the paper until the night before, but that should be the extent of your dickishness).
4. The paper is poorly done and the results, if taken at face value, will do real harm to people, particularly those from marginalized groups:  In this case, it is ok to firmly and politely destroy the paper for shoddy craftsmanship.  You can do so in a professional manner in steps 2 and 3. You’re still not being a dick, but you don’t have to frame things as questions to think about but as real methodological problems.   It’s ok to throw around the terms “dangerous” and “needs stronger proof”.  It’s a shame that there are still guys (and the occasional woman) who write papers with sexist/racist agendas who ignore basic science in order to prove that wealthy white men are superior and deserve their privilege, but there are.  They shouldn’t be allowed to do bad science.

Academic readers– is this about right?  What things are the same or different in your discipline?  Any other tips?

How to do a powerpoint presentation (social sciences, economics)

I LOVE me some powerpoints.

Think about what you want your audience to take away.  Use the rule of 3 to emphasize those points (say what you’re going to say, say it, then tell people that you said it).  Depending on how much time you have you won’t be able to get through every point in the paper, so think about what subset you want to present, what slides you want to keep in case of questions but not actually present, and so on.

Use the powerpoint as a guide to remind you what to talk about, so brief bullets/phrases instead of full sentences.  Do not read off the slides.

Some people will only want to read your slides, some people will only want to listen to what you say.  Make sure that people who do one or the other will still get the gist of your presentation.

Make sure your fontsize is big enough that the people in the back can see it if they’re wearing glasses.   My heuristic is to not go below 28 point Calibri if it’s something I want them to read.  (Table notes can go smaller)

Graphs are often more compelling than regression output.  (But keep the regression output as a backup)

Don’t use fancy wipes/fade-outs/etc.  Anything that distracts without a purpose is useless.

Development economists, behavioral economists, psychologists, antrhopologists, etc. use a lot of photos/pictures/drawings and occasionally movies.  Do that if it is common in your field.  If it isn’t, then only sex it up like that if it helps improve understanding.

DO NOT USE PREZI.  Or if you do, use it like you would Powerpoint or Beemer.  You do not want to give members of your audience migraines.

I have often found it helpful to have different versions of the same information in the powerpoint that I can skip over depending on how pressed for time I am.  So I will have a pretty chart, regression output, and summary bullets (or two out of the three) and I will use combinations of one or two of these depending on how much time I have left.  It is also helpful to know which sections can be skipped without losing the main themes of the presentation.

Practice your talk.  Know how the talk is going to differ if questions are allowed vs. no questions being allowed.

It is better to go a little under than a little over.  It is better to skip parts than to talk so quickly nobody can understand you.

Join us next Tuesday for:  How to write a powerpoint discussion(!)

Academic readers– is this about right?  What things are the same or different in your discipline?  Any other tips?

Who are your favorite authors of color?

Excelsior Bev recently asked her students who their favorite African American authors were, and we thought that was a fun question, but that we’d broaden it a bit.

#1:  Alexandre Dumas (Jr) hands down– though I didn’t know he was black until recently!   He’s not so great with his female characters (who are either paper dolls or evil villains), but his books are so much fun that I forgive him.

After that I know there are a lot of worthy POC authors who write amazing award winning serious fiction (and I did like Roll of Thunder, Hear My Cry and Their Eyes Were Watching God, but as worthy books, not fun books), but I really like popcorn books.  I really do.   So that means people like Lisa Yee and Justina Chen.  I also love almost all of A. Lee Martinez’s books.

Scalzi had a post the other month talking about the “read just women and people of color” challenge someone was doing, and I asked for recommendations for fun light stuff, but the only person who replied has a very different definition of “light” than I do (pro-tip:  Stephen King is not light).  That post also indicated to me that romance novelist Courtney Milan is a POC, which I didn’t know (I like her stuff!).  Recommendations for light stuff welcome in the comments!  (I did read some Marta Acosta light vampire stuff, and it was ok, but not worth owning.) (#2 owns the first but not the second book.)

#2 ZOMG, N. K. Jemisin all day long.  Saladin Ahmed.  Justina Chen Headley (again).  Y. S. Lee.  Nnedi Okorafor.  Dia Reeves.  Michelle Sagara (her stuff sometimes makes #1 cry on airplanes).  Gene Luen Yang.  I recently read Sofia Samatar’s award-winning novel and liked it.

And, as everybody should already know, Octavia E. Butler is objectively one of the best science fiction authors of all time.  (But not light!)

Start there!

Of course, we’re of a couple of minds about these segregated lists.  Well, not really.  It’s just a nuanced stand.  We hate the need for these separate lists and we wish that people would be included on the regular lists of “best of” because many *belong* there.  However, society isn’t there yet, so these lists are a way for people to broaden their horizons so that they can come into contact with amazing authors they wouldn’t normally read.  Being on one of these segregated lists should in no way preclude someone from going on the more general lists of “best of” and we should think really hard when we make a general “best of” list about composition to make sure we’re not running into implicit biases.  A standard procedure is to think about the best POC or female etc. author not on the general list and to compare him or her to the worst person on the general list (iterating to the next underrepresented person etc.).  More often than should be the case, that person really belongs on the general list too and was not included because of subconscious biases.  Eventually, thinking about people from underrepresented groups while making the list rather than after the list is made becomes more automatic.

One place where there are plenty of authors of color is the banned books list.  Boo.

Got anyone else we should read?  Spend your tax refund on books!  Or save it and use your library.

compound interest

One of the things Dave Ramsey is infamous for is making the claim that the stock market returns 12%/year.  Lately he’s been saying well, his money market funds return that.  (Uh huh.)  Then he goes through an exercise showing how much money you’ll make if you put away X, assuming an interest rate of 12%/year.  It’s a lot.  Because of the magic of compounding.

And obviously that 12% number is garbage.  (Reality is probably closer to 7%.)  However, after he makes this claim, he’ll often say, “Even if I’m half wrong…that’s still a lot of money.”

The implication being, if the interest rate is closer to 6% you’ll end up with half of that huge number he just calculated, which is still a huge number.

Of course you don’t.

Because compound interest doesn’t work that way.  As time goes on tiny differences are magnified with each compound, so that 6% difference starts out as half as big, but ends up compounding over time to something much larger than half as big.

Here’s a calculator because that’s more fun than doing the math by hand.  (Or at least it’s more fun than either typing out the formula or typing out the derivation of the formula.)

Take a Principal of 100,000.  Don’t add anything to it.  Assume a 12% interest rate that compounds once per year for 30 years.  You get $2,995,992.21 .  Or almost 3 million dollars.

Now let’s assume it’s actually half of 12%, or 6%.  If you’re thinking, I could totally live on 1.5 million dollars… you probably could.  But a 6% interest rate over 30 years only gives you $574,349.12, or half a million dollars.  Not chump change, but not enough to live through retirement on, even assuming these are real interest rates (putting things in today’s dollars instead of tomorrow’s inflationed dollars) and not nominal (if you assume 2% inflation, the real interest rate is 2 points lower than the nominal rate).

Half the interest rate compounded doesn’t result in half the earnings, but instead far less than that.

Losing just 1%, for a rate of 11% gives $2,289,229.66, which is a loss of about 700K!

The truth is that compound interest is magical, and the longer your time horizon, the less you need to put in to get big numbers out the other end.  However, it’s not quite as magical as a 12% interest rate would have you believe.  If Dave Ramsey is 50% wrong, you’re much worse than 50% worse off.