Null effects are fine, but you need to discuss power!

I like the way that a lot of social sciences are starting to push for publishing more studies that tried something plausible and then found nothing.  Null effects papers tend to be difficult to publish, which leads to publication bias, meaning you’re more likely to find something spurious than to not find something.

BUT.  One almost sure-fire way to find no effect is to have a sample size that is too small to pick up an effect.

If you find no effect, you need to discuss sample size.

If you find no effect from an experiment, then you really need to talk about the power analysis that you did *before* you ran the experiment that shows the sample size you would need to find an effect size.

And if you have a large magnitude that just isn’t significant, that isn’t as convincing as having an insignificant small magnitude or, even better, a small magnitude that flips sign depending on specification.

As the great Dierdre McCloskey says, statistical significance is not the same as oomph.  Or as I tell my students, meaningful significance is not the same as statistical significance.

A true null effect is something that has a small effect size, whether or not it is significant.  And if you find an insignificant null effect, then you have to discuss whether this is a true non-finding or if you just didn’t get enough observations.

Got that?  Null effect = fine, but it has to be a real null effect and not just a bad study.

Advertisements

Ask the grumpies: What do you think about that horrible Wendy Williams and Stephen Ceci article?

Taia asks:

I read your blog occasionally and am interested in your comments on this article studying hiring preferences for male/female academics in science fields.
http://www.pnas.org/content/early/2015/04/08/1418878112.abstract

Anything for an occasional blog reader?

There’s already some great commentary on this terrible article (shame on PNAS for publishing it!) <– scroll down in the link for a bunch of linked studies.

In addition to all of the problems already illuminated in the linked criticism, but there are some elements of the survey design right off the bat that have been shown to decrease predicted discrimination.  For example, comparing two functionally identical resumes next to each other results in decreased implicit bias (according to a much better written PLOS One article).  That’s why good lab studies will compare across participants rather than within participants.  Field studies do often compare within participants (job openings), but they aren’t the only two resumes being considered, and even so, a new working paper by a researcher (David Phillips) from Hope college shows that by sending functionally identical resumes in these field studies, matched pair audit studies do change how the resumes are perceived.

Also the quality of the candidates matters– there seems to be a winner take all thing going in many stem fields so when women are at the top of the distribution they’re preferred, but when they’re not at the top, they are discriminated against compared to similar males.

Finally, even if the research designs were externally and internally valid (which they are not, see linked commentary), there have been at least 19 studies showing the opposite of this study, so it’s unlikely, but these results could just be random.

(That’s not even including the history that the authors have of doing bad science to support their demonstrated agenda.)

Crowd sourcing data: An alternative to mindless games

Bored over the holidays?  Want to do something mindless that gives you a small sense of accomplishment without leaving your electronic device?  What if it also helps SCIENCE?

Zooniverse is a citizen science project– basically (if I’m understanding correctly) there’s this software that researchers can get from github that they can use to crowd-source data entry.  To be on the zooniverse page, they have some kind of competition for Science and the best projects get to go up.  People work on small bite-sized chunks of project in their spare time, essentially for free.  You can classify pictures of universes or tell them when a picture has an animal in it and what kind of animal and read histories of ancient peoples.  There’s projects to study climate change and history and wildlife.  Difficulty ranges from things a three year old can do (ex. identifying animals on the Serengeti) to reading handwriting or transcribing ancient Greek.  There’s even sounds to categorize.

Anyhow, if you need to get addicted to something on the internet, we suggest trying this out.  It might be fun!

Disclaimer:  Zooiverse doesn’t know we exist.  One of us did meet a nice gentleman who is affiliated with one of the projects though.

Have you participated in crowd-sourcing for science before?  Are there other sites you would recommend?

Finding what interests me in a new career

One of us is job-hunting after quitting academia and moving to paradise.  I have been looking for jobs I want, but I haven’t been finding that many to apply to–I still have enough resources at this point to be able to focus my search on jobs I would like rather than taking any job.  I have applied for about 20 things and gotten 1 phone interview and no in-person interviews or offers.

What do I want?  I want something sciency and researchy, in the social sciences.  I am not a clinician and not a certified CRA.  I am not a biologist or pharmacist or engineer and I do not use Hadoop (I could learn if I had to but it doesn’t seem necessary now).  I don’t program (other than several standard social science statistics softwares and some dabbling in things like .html etc., but not like C++ kind of programming) and I don’t want to.

I have [#2: excellent] skills in data analysis, writing, editing, literature review, and many things about the research process [#2:  I fully vouch for these– she reads every paper of mine before I send it out and she’s helped me a ton when stat-transfer fails me, and more than once she’s saved my rear end doing last second RA work when I was up against a deadline and I found a SNAFU.  I’ve also shamelessly stolen a lot of her teaching stuff, but that’s probably irrelevant since she doesn’t want to adjunct or lecture.]!  (See the second table below)  I can do tons of research.

I am not an extrovert and interacting with people most of the time drains me, but I interact quite successfully in teams and research groups.  I’m not interested in being a manager of people in a pure managerial sense, though I can do some and I am experienced supervising teams of research assistants.

Ever since I was a little kid, every “career interest” test I have ever taken has always come out that I should be a professor, and it still does.  However, nope nope nope!

I played with this online thing for scientists and it was kinda enlightening.  It tells you, among other things, about what your values, skills, and interests are in a career.  Here are mine.

First, here are my values of things that are unimportant and important to me in a new career:  (for these big tables, click to embiggen).  I know this is a lot to ask for, but it represents the ideal.

Second, here are my scientific skills, what I think I am good and bad at:

Third, here are my interests:

The jobs it suggests for me include faculty at a research university (nopenopenope) and the things I am already applying for, such as research manager stuff.  I would be happy to manage someone’s lab, although I can’t put up with a job where the ONLY thing I do is make other people’s travel arrangements.  I could do quite a good job in something like research administration, if it focused on compliance and not budgets (though I can and will do budgets so long as it isn’t the *only* thing). I am good at teaching but I will never do it ever again.  I love collaborating with other scientists but am not crazy about managing people.

I would like to work for a nonprofit or the VA (which keeps failing to hire me over and over).  I’m not against working at a for-profit company though, especially if the pay is good and the work is interesting.  Program-analyst type stuff seems to be a title I come across a lot for job postings.  The site also suggests that I be an epidemiologist (interesting but I’m not trained for it), a clinical diagnostician (not trained for it and don’t want to be), and a teaching faculty (NOPE NOPE NOPE).  I would be fine as non-academic staff at a university.  I do not do drug testing, nor do I have any wet-lab skills.

You can be sure that my cover letter and resume are shiny, personalized, revised, and proofread by #2 [and, #2 notes, more importantly, the career office at her former grad school went over her resume when she did the change from cv to resume].

I’m not expecting to go in at the highest level, and I don’t really want to. I am definitely willing to work my way up to some extent, but not all the way from the proverbial mailroom. My retirement funds are anemic and if the job is really poorly paid, it might be more profitable to spend that time searching for a better job, rather than being tied to a job that’s both low-paying *and* boring.

Mostly I’ve been applying for jobs that I find on Indeed.com.  But I need to expand.  And yes, I know I should be networking more (and I swear I am networking!)– this post is part of that effort.  ;)

I promise I’m not as much of a special snowflake as I sound like here; I have skills that would really help an employer if only I could convince them of that [and, #2 notes, if she could find more job openings, preferably before they’re advertised…].  Help!

Grumpeteers, what say you?  How can I get a job that pays decently and is also suited to my skills, interests, knowledge, and background?  

Top 20 baby words

DC2 is about to the age in which ze starts saying things, so I got to wondering what are the early words that babies say.

Fortunately, there’s research on this topic.  I came across a 2008 article from some psychologists at Stanford that includes a chart titled, Rank-Ordered Top 20 Words for Children Who Can Say 1–10 Words on CDI and Percentage of Children Producing Them, by Language

It’s Table 4 if you click that link.  They include Hong Kong and Beijing’s words as well.

Here’s the words for the United States (copied from Tardif et al. 2008).
(n = 264)
Daddy
Mommy
BaaBaa
Bye
Hi

UhOh

Grr

Bottle

Yum Yum

Dog

No

Woof Woof

Vroom

Kitty

Ball

Baby

Duck

Cat

Ouch

Banana

My first word (not counting Ma’s and Da’s) was the same as my oldest’s first word, “Hi” there on the list.  DC2 hasn’t gotten to “Hi.”  Months ago DC2 was saying key (for kitty) but that seems to have dropped out of the lexicon and has been replaced with Ca (for cat).  Dog has been added.    Ze says, “Yeah,” a lot to signal agreement. Ze can make three different sounds that dogs make — “bowwow” they taught at daycare, “woof” I taught hir, and DH taught hir panting [update:  ze can also make stuffed dog make the slobbery dog kisses sound now, so that’s 4].  Occasionally we’ll hear a ba for bottle, or a bana or nana for banana.  Ze may be saying a lot more, but it’s awfully difficult to tell with the pronunciation.  I remember that DC1 was really into animal sounds, especially barn animals, when ze started to talk in earnest.

Do you have any cute baby word stories?  What was your first word?  Were your first or your children’s first (if applicable) on the list?

This is why we can’t have nice things.

This [grant thing] that [redacted] has is really stupid.  So much bad science to “further women and minorities”.  Reading through their annual report and it’s thing after thing of, “We had this workshop, but nobody came.”  They’re also not checking to see if anything works even when people do come.  There’s not even data collected before and after to see if there’s even a change, much less a treatment effect.  There was one thing where they’re like, “we were going to do this survey but…”  They sent the report to me to evaluate, but the entire campus was “treated” and uh… the treatment seems to have been nothing.

Bad science makes the baby Jesus cry.  Poor baby Jesus.

They seem to have a lot of meetings too.  So basically, trying to further the careers of women and minorities at this school consists of making them go to pointless meetings.

See, this is why women and minorities can’t have nice things.

Argh!

(Note:  Some details in the above rant have been changed to protect both the stupid and our own rear ends.)

Are you ever astonished by the amount of bad science done for a good cause?  Have you ever noticed that it’s always the under-represented who have to waste time in meetings?

The negativity jar

The Imposter Syndrome and other forms of negativity can keep people, especially women, from achieving as much as they should.  If you say enough negative things about yourself, eventually other people start to believe them too.

One of the things that we did back in graduate school (during the job market) was have a big jar named the “Negativity Jar.” Anytime we said something negative about ourselves, we would have to put a quarter in the jar (we were poor graduate students– you might want to up that to $1 or $5). That forced us to restructure to say things that were actually true– to get at what was actually bothering us, and not to reinforce the negative lies. It forced what Cognitive Behavioral Therapists call “Cognitive restructuring.”

After about 2 weeks there was no more money put in the jar. At the end of the year we were able to buy a little bit of chocolate, not the hard liquor we’d been planning on.

Have you ever had a problem with negative self-talk?  What have you done to address it?  Did it work?