In which I critique — ah, the hell with it.

I have been known to critique science before.  Sometimes things on the internet make me want to critique bad reasoning.

But THIS? The mind boggles.  ESP, really?  I wish that I could read the reviewer comments on this paper.  I boggle and become incoherent.  Apparently in some circumstances, college students can detect the appearance of porn ahead of time, very slightly better than chance.  Also:

From the NYT article:
“In his version, Dr. Bem gave 100 college students a memory test before they did the categorizing — and found they were significantly more likely to remember words that they practiced later. “The results show that practicing a set of words after the recall test does, in fact, reach back in time to facilitate the recall of those words,” the paper concludes. “

WHAT?!?!?!?!???

#2 chimes in:  note this telling bit in the NYTimes article:

So far, at least three efforts to replicate the experiments have failed. But more are in the works, Dr. Bem said, adding, “I have received hundreds of requests for the materials” to conduct studies.

Remember that within a 95% confidence interval, when you perform this experiment 20 times, one of those times it is going to come up significant, on average.  Even if he hasn’t falsified anything or added any other kinds of accidental cues, chances are every once and a while the wrong result will come up statistically significant, even when there is no actual relationship.

11 Responses to “In which I critique — ah, the hell with it.”

  1. First Gen AMerican Says:

    Data are dangerous. People generate data and then draw conclusions but there is a big difference between an experiment and a meaningful statistically significant experiment.

    Since I work with design engineers I deal with data all the time and I’m constantly telling them that just because our test data says one thing doesn’t mean it’ll perform exactly the same in their design or environment. Until you test the exact conditions, you’re just making an educated guess. And yes, there are special cause effects that aren’t always obvious when a bit of data gives you an unexpected result and it’s not always apparent what the root cause is.

    People assume that data is gospel but there is a lot of room for error so that’s why people replicate experiments.

    So, in short, I concur. I don’t know why these types of goofballs even make it in the news.

  2. everyday tips Says:

    I don’t even know if I understand this. Maybe if I think about it later, I will remember it now?

    • nicoleandmaggie Says:

      So basically they did a small study and that students were statistically significantly able to predict when porn was being watched in another room.

      #2’s point is that if you flip a fair coin enough, you’re going to end up with a few sets of trials where it seems like the coin is unfair, just by random chance. Whenever you do statistics, if you have a 95% confidence interval, you will be wrong 5% of the time (on average).

  3. Comrade PhysioProf Says:

    My understanding is that this paper was specifically designed as a reductio ad absurdum of the fallacy of considering statistical significance a sufficient threshold for establishing that an effect is real.

    • nicoleandmaggie Says:

      Huh. I didn’t get that at all. But you may have inside information.

      It would be encouraging if that was the truth.

      But:

      “What I showed was that unselected subjects could sense the erotic photos,” Dr. Bem said, “but my guess is that if you use more talented people, who are better at this, they could find any of the photos.”

    • Dr. Koshary Says:

      I seriously hope that CPP is right about this. But still, it’s irritating as hell that NYT prints this with a completely po-faced “gee whiz” attitude. If I had to guess, Bem is doing this to illustrate the point that CPP makes, and he’s having fun staying in character, lest he give up the game too soon. He doesn’t sound like the kind of researcher to just wig out and start talking about ESP and third eyes and whatnot. I have the feeling that the reporter couldn’t make head or tail of any of this, and just waited for people to provide quotable summaries of everything.

      Oh, and hi!

      • nicoleandmaggie Says:

        I cross my fingers that you are right. I didn’t know Bem was an actor or anything, but I guess we’ll see! At the very least he provides a valuable exercise for students to critique.

  4. Roshawn @ Watson Inc Says:

    From your article critique, this appears to be classic type I error. What part of the 3 failed replication cohort suggests otherwise! However, if what Comrade PhysioProf said is true, this would be way more interesting.

  5. Ask the Blogosphere « Grumpy rumblings of the untenured Says:

    […] Daryl Bem, about whom I have posted before, attempts to explain his concept of time-traveling pr0n by invoking quantum physics: […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: