Selection into the sample on the variable of interest: A Rant

First, a disclaimer:  I dislike mike the mad biologist.  I blame men like him for Trump being president because of his relentless attacks on ‘this woman, I would support any woman for president, just not this one” and the fact that he’s one of those assholes who does absolutely zero work and again, attacks people who are actually doing something as always doing the wrong thing.  Those men make doing things that they claim to support more difficult and emotionally draining while not lifting a finger themselves.  The only reason I ever look at his blog is because he’s on Bardiac’s blogroll and occasionally he’ll have one of those post headlines that I have to click.  Previously I thought he was just an unselfaware misogynist blowhard.  Now, I’m realizing that he’s … also not got very good statistics training.  (Interestingly, I’ve seen a recent survey that finds that people who are explicitly racist also just tend to be wrong about other things unrelated to their racism.  Not saying that translates to implicit misogyny, but… )

Ok, so here’s the post in question.  In it, he claims that a survey of people who aren’t getting vaccinated proves that time pressure and inability to take a day off work are not reasons.  (Therefore any policies targeting making getting vaccinations easier or getting time off work are wasted effort.)

The problem?  If you click on the survey, it says it has a 6.1.% weighted rate for taking the survey.  (So only about 6 out of every 100 people they sent the survey out to actually responded.)  Also, it is an online survey.

If that 6.1% were a randomly selected sample of a general population, there wouldn’t be a problem.   The problem is when the selection into the sample is based on the outcome that you’re measuring.  In this case, if you’re measuring people who don’t have time or ability to get vaccinated, well, likely they don’t have time or ability to take a survey either.

The Census Bureau isn’t stupid– they know this is a problem and they have lengthy documentation about the non-response bias in the sample generally.  They make it clear what you can trust the results for and what you can’t, as well as the limits of their weighting schemes.  The survey isn’t completely useless, but it is only externally valid for the groups that were surveyed!

I had been planning to use this little example of “sample selection on the Y variable” in my stats class this fall, but now I can’t because his response was so ironically ignorant that I have to blog about it instead.

Here’s his response:

The low income people who are supposed to be burdened by the time constraints also don’t report access as an issue compared to other factors. Are there any data that could convince you, or will the answer always be the same?

So– I guess I was right about his complete lack of self-awareness.  Can you imagine being convinced to make a huge policy change based on one extremely selected survey?  The people who ran the survey would never ever want you to make a policy change based on this result!

The answer to the question of what would be convincing (taking it seriously rather than just an accusation of me being set in my ways):

1.  A nationally representative sample that found the same result.  The US government has some of these, where you are required to take the survey and they have much better response rates (not perfect, but much better).  This survey is not one of them.
2.  A sample representative of the population who we are trying to target with our policies (ex. a state going into a few factories with onsite vaccination clinics before expanding the program).
3.  Multiple biased surveys that are biased in different ways (that is, are not biased on the outcome variable of not having enough time).

No good policy maker would make policy based on such flimsy evidence as the survey mike the mad biologist presents.  In fact, we rarely make big changes based on the result of any one study, unless it is the only study available or is the only well-designed large experiment available.  And even then, good policy makers keep their eyes out for new evidence and try not to do huge national things when the evidence is scant.  Ideally we’ll have a largescale randomized controlled trial, but failing that we’ll take a series of mixed methods– qualitative information, event studies (these two are the easiest and cheapest to do but can be biased depending on how they’re done), natural experiments, and so on.  Ideally we’ll have information about heterogeneity– we think, for example, that the effects of the Affordable Care Act and the effects of universal health insurance were different for Oregon compared to Massachusetts compared to Wisconsin or Tennessee.  And that could be because they have different populations and different starting environments, or it could be that each of these states had a different methodology used to study it with different biases.

Unlike Mike the Mad Biologist, every single thing I do (in research and in teaching!) has the potential of helping or harming someone’s life.  I have to be extremely careful.  I don’t make policy recommendations until the bulk of evidence supports those recommendations.  Because, getting back to that first disclaimer– I’m actually out there doing stuff, not just complaining about the people who do things.

So yeah, I teach my students about how not to use samples that are selected on your variable of interest.  It’s a more challenging concept than people say, lying about their weight or height, but it is an extremely important one.  I have a lot of students who go out and design/make/evaluate policy when they graduate.  Hopefully the lessons I give them remain with them.