Transcript

(This is an automatically generated transcript)

Fraser Cain [00:01:34 ] Astronomy Cast episode 712. How peer review fails. Welcome to Astronomy Cast, a weekly facts based journey through the cosmos. We help you understand not only what we know, but how we know what we know. I’m Fraser Cain and the publisher of Universe Today. With me, as always, is Doctor Pamela Galea, senior scientist for the Planetary Science Institute and the director of Cosmic Quest. Hey, Pamela. How are you doing? 

Pamela Gay [00:01:57 ] I am doing well. We’ve talked in the last few weeks about the signs of spring. And as you and your your wife record them based on things coming on outside. I’m not sure if last week was either first, daffodil or first stunting. Right. But I’m afraid it was first Skunk King two months earlier than normal. And and yeah, the skunks are out of torpor and I has a sad. 

Fraser Cain [00:02:27 ] Oh, is that because your dogs are going to get into skunk trouble? 

Pamela Gay [00:02:34 ] Oh, we already have. That’s the thing. The the Stella ran out the back door, made it all of about six feet away from the house. Got skunked, which meant the house also got skunked, right? And ran back in foaming, as they do when they take it straight in the mouth. And, yeah. 

Fraser Cain [00:02:54 ] Do you think they’ll learn this lesson? 

Pamela Gay [00:02:55 ] No, no. Well, honestly, it was after dark and she didn’t even make it all the way down the dog ramp, so I don’t think she had a chance to see the skunk before she got blasted. 

Fraser Cain [00:03:07 ] Right, right. This guy was just there. Yeah, yeah. 

Pamela Gay [00:03:12 ] Not the dog’s fault, right. That time. 

Fraser Cain [00:03:15 ] Yeah. Well, what’s the recent one that we have? Mallards arrived at our ponds is a thing that. Like that? Yeah, it’s pretty cool. And herring season, I guess is the big one, which I mentioned this before the show that we have this, the Pacific herring spawn, which is the largest biomass migration on planet Earth. And so we see whales and eagles and sea lions and seals and it’s it’s mayhem. 

Pamela Gay [00:03:48 ] It’s amazing. 

Fraser Cain [00:03:49 ] Thousands of seagulls on the side of the beach for flying in these giant spheres. Bait balls where they’re, above where the herring are spawning. It’s it’s amazing. It’s incredible. So, yeah, it’s a pretty dramatic one, I would say, like, probably the biggest nature based event of the entire year is happening right now here. I think. 

Pamela Gay [00:04:10 ] Wild. 

Fraser Cain [00:04:12 ] You have probably heard the the best kind of science is peer reviewed research published in a prestigious journal. But peer review has problems of its own. So let’s talk about that today. All right. What is peer review? 

Pamela Gay [00:04:26 ] It is the process by which a scientist submits a paper to a journal that has references and research and describes their methods and describes their observations, and comes to some conclusions or describes their theoretical modeling and their conclusions. And then the journal that they submitted it to sends the paper out to one or more other PhD researchers to go over the paper and to assess whether or not the conclusions they reach and the way they took their data is sufficient and valid, that the paper can be published such that other people should trust the results are a logical conclusion based on what occurred. 

Fraser Cain [00:05:17 ] Right. So it’s like a double checking that what you’re saying in the paper is supported by the evidence. Yes. Right. And I think when we have, when we think about peer review. As a as a journalist and was spent some time being an editor. I think of it as roughly analogous that that somebody submits their article to me. I go through the article line by line, sentence by sentence. I fix the grammar. I also note places where I think it could be tightened up, where they are making claims, they are providing quotes there and then if you’re like a like next tier, if you’ve got more resources, if the New York Times, then you have a fact checker who goes in, confirms that that somebody actually said this quote, right, that it is a comprehensive and exhaustive process to make sure that in my case, the article is well-reported and grammatically correct, and Ben fulfills all this journalistic requirement. And if you take that analogy, move it over to research. You’re you’re like. It’s like a professor grading a paper. Where they’re they’re reading it line by line, making sure everything’s fine. Checking to make sure that the math has been done correctly. Catching any mistakes. It’s a 1 to 1, but it’s not that at all, is it? 

Pamela Gay [00:06:47 ] No. It is rife with problems at all sorts of different levels. And in this episode, we’re going to hit on a lot of the different places. Full disclosure, this episode was completely inspired by John Hopkins University press release that caused both Beth and I to giggle at our inboxes. The title of this press release is Interstellar Signal Linked to aliens was actually just a truck, right? 

Fraser Cain [00:07:20 ] Yes. Yeah, we’re reporting on that. We’re working on the story for that right now. 

Pamela Gay [00:07:23 ] Yeah, yeah, it’s it’s one of these things where we have seen two different things going on through peer review. We have seen papers that should get published through peer review, not making it through. And scientists basically flinging up their hands and saying, what’s going on here? Why won’t you publish my results? On the other side, we have things going through and getting published where it’s just like, all right, I, I don’t want to live in the universe that’s described in this paper, and I’m pretty sure I don’t. 

Fraser Cain [00:08:00 ] Right. Yeah. I think about the rat genitalia I image that was made it through peer review recently. I don’t know if you saw this story. 

Pamela Gay [00:08:09 ] No. 

Fraser Cain [00:08:10 ] So someone had had, I don’t know, gone and asked ChatGPT to make an image, a scientific image to go along with their paper. And it was hilariously, you know, with, with gibberish letters on it and a rat with a gigantic, enlarged genitalia as big as the rat. I think it was ridiculous. And it got it appalled by the it somehow went through peer review. So I guess I want to sort of shatter that, that expectation that people have that, that peer review is this careful, laborious process to make sure that the work that’s in the paper, like if you are a peer reviewer, let’s say there is a paper and you are a, you know, someone who has a PhD in the same field and you’ve been sent the paper, what is the expectation of the work that you’re going to do to fulfill your peer review requirements for that paper? 

Pamela Gay [00:09:06 ] So the general expectation is you will go through, you will read the paper, and you will make sure that the logic flows sensibly and that things that are referenced, are reference correctly and that the analysis that is done is sufficiently complete to actually prove what they’re trying to say. The running joke is there is usually a reviewer, number two, who sends you back a essay on all the things that are wrong with your paper, why you’re a terrible human being, why your mother failed the universe by giving birth to you, and how, in order for your paper to actually get published, you need to start from scratch, do a complete dissertation level analysis and also make your paper shorter. 

Fraser Cain [00:10:00 ] And accept their work. 

Pamela Gay [00:10:02 ] Yes, yes, yes, you have to cite their work rates is usually part of that. Yes. 

Fraser Cain [00:10:06 ] But but the point is like like how much time? Like let’s imagine that that somebody has given the paper to peer review. How much time are they expected to take doing their job? 

Pamela Gay [00:10:18 ] A few hours. 

Fraser Cain [00:10:19 ] A few hours, so the paper could have taken hundreds of hours of observation. 

Pamela Gay [00:10:25 ] Yes. 

Fraser Cain [00:10:26 ] 100 hours to write. 

Pamela Gay [00:10:28 ] Yes. 

<