What does statistical significance mean?

One of my students sent me this article because we spend some time in class covering Type 1 and Type 2 errors.

All the .05 threshold means is that you have a false positive 1/20 times.  A .005 threshold would say you’re getting a false positive 1/200 times.  So by moving to a .005 threshold, you’re less likely to get a false positive.  That’s good, right?  In common parlance, we’d be less likely to send an innocent person to jail.

Well, that depends.  At the .005 threshold, we’re more likely to get a false negative than you would at the .05 level.  That means we’d be more likely to get a guilty person go free.  (Indeed, the only guaranteed way to send no innocent people to jail would be to send nobody to jail.  I, for one, am happy that folks like Charles Manson are behind bars.)

It isn’t as easy as saying, oh we should just switch to .005.  When you adjust the p-value you’re making trade-offs between type 1 and type 2 error.  With a lower p-value threshold you’re going to be getting a lot more false negatives even with fewer false positives.  What we always need to be cognizant of when we’re doing policy is that significance isn’t everything– we also have to think about what the damage is if this information turns out to be incorrect.  For example, doctors recommend that pregnant women should heat up cold cuts if they’re worried about listeria, which is a very low probability event but if it happens it’s horrible.  It’s pretty easy to avoid room temperature cold-cuts for 9 months, so unless there’s some other difficulties attached to diet, women will probably follow this recommendation.  (And if one accidentally eats room temperature coldcuts while pregnant, one shouldn’t freak out because the probability of getting listeria is very low!)  But if we’re talking about something like doing chemotherapy or surgery, that’s a much more onerous action and we might want to be more sure we need it before going ahead with it.

Another thing to note is that the article talks about how physics and genetics have already made this switch, while most social sciences haven’t.  One big difference between the fields that have made the switch and the fields that have not is how easy it is to get large samples.  A larger sample size will make it so your sample behaves more and more like the population that you’re trying to study.  We can reduce both Type 1 and Type 2 errors simply by increasing the sample size.  So why don’t we do that?  Well, it turns out that increasing the sample size can be very very expensive when you’re dealing with people and behavior.  Sometimes doing the study with a large enough sample to get 80% power and an alpha of .005 might be more expensive than just throwing that same money at the intervention you’re trying to decide about, whether or not it actually works.  There probably is some resistance because people in these fields want to be able to publish their 5% results, but that’s not the main or only reason we haven’t yet made the switch.  Research is complicated and expensive and we have to make trade-offs.

The context for these really does matter, and you shouldn’t necessarily put off making policy choices just because your sample size is too small to get significance (or to make policy changes just because you have significance).  You always have to be aware of the costs and the benefits.


(Incidentally, in case he comes across this, Hi Dan!  I’m assuming that the reporter greatly simplified your arguments here because I know you must know this stuff.)


IRA advice

Disclaimer:  We are not professional financial advisors.  Talk with a certified fee-only financial planner who has fiduciary responsibility or do your own research before making important financial decisions.

Next week I’ll post a rant about SIL’s truly awful retirement options at her workplace and how mad it makes me that a non-negligible number of school teachers are getting scammed.  This week I’m instead posting the advice we gave them to put money in an IRA outside their work rather than the unmatched and worse-than-edward-jones 403(b) option that she has at work.

If the couple makes less than 99K, they can each contribute up to $5,500 to a traditional IRA (or $11K) each year.  This would mean that they would get a discount on their taxes this year but would have to pay taxes on the earnings in retirement.

If they make less than 186K, they can each contribute up to $5,500 to a ROTH IRA (or $11K).  This would mean that they wouldn’t get a tax break on what they contribute this year, but they wouldn’t have to pay taxes on the earnings in the future.

In terms of which is better– generally if you think you’re going to be in a higher tax bracket in retirement, you should pick the IRA Roth, if you think you’re going to be in a lower tax bracket in retirement, you should pick the Traditional.  It’s hard to say what the tax situation is going to be like in 40 years.  Personally we do about 50% because on the one hand we’ll be making less money in retirement, but on the other hand at some point they’re going to have to raise taxes in the US to pay down the debt and tax rates are historically low right now.  If you want lower taxes now, pick Traditional, if you want lower taxes later, pick Roth.

The best place to invest for an IRA is in Vanguard https://investor.vanguard.com/ira/how-to-open-an-ira?lang=en because they have the lowest fees in the industry.  One potential problem is that they make you start with $1000, and if you don’t have $1000 to invest right away your options are to either save up until you do have $1000 (you have until tax day of the next year) or to go with another provider.  Etrade is a good choice if you don’t have $1000.  Fidelity is another possibility.  Vanguard is definitely the easiest to work with.  I’m pretty sure that you can set-up a monthly auto-deduction from your savings account for these (but you may have to hit the minimum for Vanguard first), but it might make more sense to save up in a separate account and make the purchase all at once each year (so there are lower purchase costs).

The best investment choice is a low fee broad-based index fund or exchange-traded fund.  These are basically mutual funds where instead of paying a manager to pick out funds, just try to match the market for a broad-based set of stocks and sometimes bonds.  These will generally match the market that they’re paid into or mimicking.  If you want to do some active management, you can pick out a Vanguard stock fund and a Vanguard bond fund and get the percentages based on your retirement and then rebalance each year when you add new money.  What matters when choosing funds is not their “Returns since inception” or even the returns at all (since they don’t predict future returns and can be manipulated in ways that make the same funds look higher or lower earning based on when they start counting) but the fees.  You want low fees and broad diversification.

A much easier thing to do is to pick your date of retirement and buy a Vanguard Target-date fund.  A Target-date fund is a mix of broad-based index funds for stocks and bonds that becomes less risky as you get closer to retirement.  (You can do this even if you’re using etrade or fidelity– fidelity has their own target date funds, but they have higher hidden fees than do Vanguard’s.)  These are great because you pick one number– the year you think you’re going to retire– and it takes care of rebalancing and changing your mix of stocks and bonds for you.  Most of our retirement money is invested in Vanguard Target-date funds. It costs a little bit more than just buying the funds it uses, but a small enough amount more (~1/10 of 1%) that we’re willing to pay for the convenience.


If you make under 186K jointly, you can buy up to two $5,500 Roths (no tax break now, tax break later).  Under 99K you could instead get traditional IRAs (tax break now, taxes later).
If you have at least 1K to invest, then buy a target date fund from Vanguard to put in the IRA with the date of your expected retirement.

Link Love: The Wrong Things Are On Fire

Gentlefriends, some things are on fire that should not be (e.g., Montana) and some things are not on fire that should be (e.g., racism).  Let’s read some links about why we can’t have nice things.

Jeez, #2 sure has sent me a million tweets this week.  I’ll be serving as your personal TweetDeck.  Here we go:

Did your identifying info get pwned in the Equifax clusterfreak?  This page is for you.

Sexism is not on fire enough, for my taste.  Burn it to the ground.  Burn it all down.  Can we have nice things yet?  Probably not.  Sigh.

Hey, peep this lady’s awesome mathy accomplishments!

Double-checking… no, we can’t have nice things.  Definitely not.

I guess everyone but me cares real hard about Russia.  I care only a little, due to outrage fatigue.  Especially outrage-over-Russia fatigue.  Texas continues to be an awful pit.  (Topped only by Facebook, Reddit, etc. etc.)

Pseudo-CSA works for this blogger.  Reset your wardrobe and possibly your mind.

Questlove loves SCIENCE!
So does this guy.

If we had time enough

The world’s gonna know your name!

#1 here again.  This is completely me this week (and kind of the past 10 months).  It me.  I’m that.

Is it you, too, Grumpeteers?


Ask the grumpies: Baths vs. showers?

Leah asks:

what are your opinions on baths versus showers?

#1:  I love both, but one thing I do not understand is how OCD germaphobe John Green can wax so poetically about baths and so stridently against showers.  Does he not realize that you are wallowing in germs when you take a bath?  Plus if the bath hasn’t just been cleaned and you’re not careful about drains and things, some of those germs might belong to other people.  These concerns don’t generally bother me (well, I don’t do hotel Jacuzzis), but I totally would have guessed the opposite for John Green’s preferences.  Er hem, back to the original question.  I love a good soaking bath, but I usually take showers because they’re faster, our bathtub is so big I feel guilty using it except when I really need a hot soak, and because I’m allergic to the water in our town and we’ve got a shower filter set up but not a bath filter (we could and should probably get a whole house filter now, but previously we couldn’t afford it– the shower and kitchen sink filters combined are an order of magnitude cheaper).

#2:  I like baths but they must be in a separate tub that nobody showers in.

Notes from a 3 hour implicit bias training

Faculty and staff had mandatory implicit bias training this year.  Last time (>5 years ago) we did this it was voluntary and all I remember from it was the speaker bringing up a (female, foreign-born, adjunct) volunteer from the audience and white male full professors commenting on her clothing and appearance because the speaker asked them what their initial impressions of her were.  It was enormously cringe-worthy.  This time it was a bit better, but I still came away with the feeling that, like economics, perhaps a little training is worse than no training at all.

I think I understand now why implicit bias training has been shown* to decrease implicit bias in people who already understand implicit bias and increases it in people who don’t really believe in it.

The first audience comment was an ageist joke.  Most people laughed.  I told the commenter that was not appropriate.  If I hadn’t been there, would anybody have said anything?

The students took this training for the first time last year.  I now understand why I got comments on my course evals saying that I was micro-aggressive towards white men and favored under-represented minorities and women over said white men.**  This training is focusing on making everybody in the audience feel like victims and giving them the language to talk about that.  I work very hard at inclusion in my classes and inclusion can feel like micro-aggression to the majority who is used to feeling like they’re special.  The first example the speaker gave was an example about the speaker hearing someone using the term “redneck” and joking, “you did not just say that.”  To her credit, she noted that most of the (Southern) audience was staring at her in disbelief and asked why.  After some native Southerners pointed out that was a pretty milquetoast insult, I noted that there really aren’t any powerful epithets against native straight white men in the US.  People in the audience seemed to agree.  (They probably didn’t need me there for that one.)

During various exercises, one straight white guy after another shared anecdotes about when they felt like they’d been discriminated against or stereotyped.  So many short-haired white guy heads nodded during these recounting while the rest of us just sat there.  The speaker applauded them for their sharing and made points about how everyone is put into groups.

It went on like that.  I broke in a few times to note that thinking you’re aware isn’t enough– people don’t realize that they’re calling on men more than women– they think they’re being equivalent.  They think 35% is 50%.  So you really do need to keep track of who is talking, or (as another professor suggested) you need to randomize cold-calls.  I talked about how to make cold-calling less scary and how to include more students, even those who are silenced.  I talked about other techniques that can be used to make groups more inclusive.  Having good intentions isn’t enough.   But thinking it is enough is dangerous.

There was a lot of talking about problems, nothing about solutions.   The speaker brought up examples of incidents and asked if we’d seen them and to discuss them (and how they make people feel), but didn’t talk about possible bystander reactions.  There was no discussion of relative difficulty, no checking white guy privilege.  Most of the exercises had the purpose of making people understand what it feels like to be discriminated against… but, as I said before, for people who aren’t actually discriminated against, not being treated like princes feels a lot like discrimination.

I suspect there’s implicit bias training that works better than what most universities are presenting.  This is not yet a solved problem.  What can be done in a 3 hour lecture hall, even with group exercises?  I don’t know.  But my other colleague who has studied this a lot for that university-level committee we were on thinks that maybe not trying to cover everything and instead focusing on the major problems affecting our students and our faculty right now according to the latest campus climate survey (islamaphobia, racism, homophobia, gender discrimination, or some subset thereof) and providing solutions on what to do for various instances might be the way to go.  If these were smaller sessions, maybe the IAT (though again, its use has had mixed results depending on how receptive the participant is).

Have you seen implicit bias training that actually works?

*too lazy to look up the citation, but it featured heavily in a university-level committee I was on

**fairly sure I’m not micro-aggressive towards white men.  However, I am intentionally micro-aggressive (as well as explicitly “you coming in late is disruptive stop doing that”) to people who wander into class late, and last year only white men wandered in late.  Most white men did not wander in late.

Cranking through cookbooks again

Long-time readers will know that #1 gets the bulk of her excitement in life from food.  While some people enjoy eating the same (excellent) ~14 meals on rotation, I am too much of a food dilettante.  On top of that, we live in a relatively small town, so even going out can’t bring excitement to my life because we’ve already had everything our town has to offer until places go out of business and get replaced with the new crop of restaurants.  (And our latest and best beloved CSA went out of business a few months ago, so no being forced to try new veggie things.)

Which means that cook-books are a lifeline.  Yes, the internet is great, but the internet takes effort if you want to find something *new* and *different*.  It’s easy to use the internet to say, find the “best chocolate cake recipe” but not so easy to find something that you don’t yet know exists.

Some cookbooks are really amazing.  Here’s a list from 4 years ago of cookbooks we have loved.  I love taking a cookbook that is ~100% winning recipes and just trying them all, even if some of them sound a little weird (example:  egg and onion soup from Help! My Apartment Has a Kitchen! turned out to be quick, simple, and delicious, much to our surprise).

Recently I’ve been on kind of a new American/comfort food kick.  (Part of this is because DC1 has started being a pickier eater for no good reason and DC2 has responded by being unable to handle even the smallest bit of spiciness.  American tends to suit both palates so long as we skip cheese and tomatoes.)   I just retired the Better Homes and Gardens 10 years of best recipes book I’d been digging through after realizing we had marked every recipe we’d tried from that book in the number of years we’ve owned it with “ok, nothing special” except for their cake recipes and a single chocolate chip cookie recipe (the other cookie recipes we tried all say, “meh” or “too cakey” or “nothing special”).  Better Homes and Gardens has good cake, but we don’t make cake that often.  Now I’ve dug out the Cooking Light book with the same theme– 10 years of five star recipes.  So far it’s been giving us better luck, especially when I cut down on sugar and switch out the non-fat ingredients with full-fat alternatives.

It’s possible that we need to get more kids’ cookbooks.  The Disney Princess Cookbook has surprisingly good meals, but not very many of them.  Kid Chef has more difficult recipes, but they’ve almost all been winners (we weren’t that crazy about the sesame bar cookies, but there are a number of recipes that were so amazing that they made our “make for other people” list).  We’ve had the kids’ fun and healthy cookbook for years and it’s been a reliable go-to.

Because DC1 is the pickiest eater, zie is now in charge of menu planning and we have been pushing hir to do more cooking (today the kids made monkey bread from the Disney cookbook… it uses an excellent buttery biscuit dough for the balls which are then dipped into butter and cinnamon sugars)… so I hope I’m passing some of this cooking excitement on to the next generation.  Maybe no-knead bread will be more enticing than drugz for them too.  ;)

What cookbooks do you think are worth cranking through?  Where do you find new recipes?  How do you deal with getting out of a cooking rut?  (Or do you prefer repetition?)

Link Love

Sorry we’re late again.  My Thursdays and Fridays are now packed with people so all my introverted self wants to do when I get home on Friday is go to sleep after spending time with the family.

Lots of articles this week (that we’re not linking to) about Trump doing X, Y, or Z.  Like going with the democrats to only raising the debt ceiling for 3 months instead of the later time the Republicans want, or the 6 month deadline for congress to do something about DACA recipients.  All of this seems to me to be part of his reality show training.  No, he isn’t rolling over for the dems.  He’s making *the most dramatic choice*.  He’s doing the reality show thing where he’s introducing man-made drama.  In this case his “apprentice” contestants just happen to be congress being given an artificially shortened time-line to work miracles.  This isn’t grand strategy.  It’s what makes the kind of reality television I hate.  And he’s dragging all of us with him.  I’m waiting for the Atlantic article on this, but instead they seem to be responding to each new thing separately and trying to figure it out from like logic or something.  The logic is TV drama.  (Also, of course, there’s the racist thing– most of his decisions are just flat out racist and evil.)

In other news, Equifax allowed my account to be hacked.  Now it is trying to make money off of me.  Fortunately it looks like after some intervention by the NY attorney general, I won’t be losing my right to sue if I take them up on their credit monitoring.

Some information about TX abortion services you can donate to.

I feel like Sim’s permanent place in CITI training about the Tuskegee study is a better reminder of what he did than is a statue.

What to make of the new facebook-Russia revelation

Even more emoluments

man who has it all requests token male historical figure suggestions for a friend’s syllabus

EPA now requires political aide’s sign-off for agency awards, grant applications

Paramedic who rescued Harvey victims may be deported

The eternal banality of the white-dude pundit

Libbra Bray has thoughts on the Lord of the Flies remake

Try this learning hack to help students remember more

Personal finance resources

A floor too far

Sweet romantic story in the commentary

Congratulations, crazymama, Phd!

Just netflixed this pre-code movie that I found when I was looking for Edward Everett Horton movies.  (He was in a LOT of pre-code stuff.)  This little scene online (see below) made me want to watch the entire movie and boy is it worth it.  I normally dislike movies in which all of the main characters are horrible people, but this one is so clever and so sneaky with its social commentary that I can do nothing other than heartily recommend it.