<iframe title="Embed Player" src="https://play.libsyn.com/embed/episode/id/31966957/height/128/theme/modern/size/standard/thumbnail/yes/custom-color/87A93A/time-start/00:00:00 /playlist-height/200/direction/backward/download/yes/font-color/FFFFFF" height="128" width="100%" scrolling="no" allowfullscreen="" webkitallowfullscreen="true" mozallowfullscreen="true" oallowfullscreen="true" msallowfullscreen="true" style="border: none;"></iframe>

Do policies built around social and behavioral science research actually work? That’s a big, and contentious, question. It’s also almost an existential question for the disciplines involved. It’s also a question that Megan Stevenson, a professor of law and of economics at the University of Virginia School of Law, grapples with as she explores how well randomized control trials can predict the real-world efficacy of interventions in criminal justice. What she’s found so far in that particular niche has echoed across the research establishment.

As she writes in the abstract of an article she saw published in the Boston University Law Review:

In this Social Science Bites podcast, Stevenson tells interviewer David Edmonds that “the paper is not saying ‘nothing works ever.’ It’s saying nothing works among this subset of interventions, and interventions, as we talked about, are the type of interventions that get studied by randomized control trials tend to be pretty limited in scope. You can randomly allocate money, but you can’t randomly allocate class or socioeconomic status.”

Despite this cautionary finding in her research. Stevenson hasn’t despaired about her career choice or that of other social and behavioral scientists. “Many of us are in this line of work because we care about the world,” she notes. “We want to make the world a better place. We want to think about the best way to do it. And this is valuable information along that path. It’s valuable information in that it shuts some doors. … So keep trying other doors, keep experimenting.”

To download an MP3 of this podcast, right-click this link and save. The transcript of the conversation appears below.

David Edmonds: This podcast has now been running for many years. Over that time, we’ve heard from a number of social scientists about interventions designed to solve various problems. For example, one of the early Social Science Bites was with Lawrence Sherman, well known for his experimental work in criminology. In that interview, he told us how randomized control trials had helped make some important discoveries; hot spot policing, which concentrates police resources in a small number of places, could be highly effective in reducing crime, or so we were told.

Recently, a new paper by Megan Stevenson, who teaches in the law school at the University of Virginia, has cast doubt on the idea that any such interventions actually work. Not surprisingly, her paper has caused a huge storm in the world of social science. Megan Stevenson, welcome to Social Science Bites. 

Megan Stevenson: Thank you so much. Lovely to be here. 

Edmonds: Today’s topic, why interventions in the criminal justice system don’t work. So, my first question, what do you mean by intervention? What interventions are we talking about?  

Stevenson: Well, in specific, we’re talking about interventions studied with randomized control trials. So randomized control trials, or RCTs, as we call them for short, are a way of evaluating how effective something is. You’re probably most familiar with them from the medical setting. You know it’s how we figured out that vaccines work and don’t have terrible side effects.

Used in the social science space, in particular in criminal justice, RCTs are a way of evaluating programs or interventions like intensive probation, like hot spots policing, that’s when you send police to a hot spot at a corner where there’s a lot of crime to see how much you can reduce crime. Programs like tough on crime, Scared Straight programs where you take a bunch of kids into prison, get them really scared about it, boot camps, things like cognitive behavioral therapy, so on and so forth. So, it’s any sort of program or policy that governments do, that NGOs do, to try and reduce crime or help people that have been involved in the justice system get on a better path. 

Edmonds: So potentially multiple interventions. You recently wrote a paper which caused, I think it’s fair to say, a huge stir in the social science world. It was called “Cause, Effect, and the Structure of the Social World.” Tell us what you set out to do in that paper. 

Stevenson: Well, this paper reviews 50 or 60 years of RCTs in the criminal justice space, and what it finds is that when you take a really broad bird’s eye view of the literature, you find that very, very few interventions, when evaluated with this kind of gold standard method, are found to achieve what they set out to achieve, and the ones that do tend not to replicate when tried again in another time or another place. 

Edmonds: I guess, in retrospect, examining these multiple studies to see whether any intervention works, it was a pretty obvious thing to do, but nobody had done it before. Was there something that triggered the thought that this was potentially a fertile area of research? 

Stevenson: Well, that’s sort of, you know, how I describe the study, but the way that it came about is kind of the opposite way. So I’m an economist. I’ve been studying the criminal justice space for 10 or 15 years, which means I do applied statistical research to see what works and what doesn’t, what’s going on right now with the criminal justice system, mostly in the United States. Honestly, this is just something that I started noticing over the years. It’s something that I think most people know in the back of their minds. Sometimes people talk about but rarely gets acknowledged. The research that people hear about, that people talk about, is the rare case of success.

You know, as social scientists, we’re programmed, or, you know, we’re incentivized to try and find the successful intervention. That’s what gets published. That’s what’s going to build our career, give your tenure, so on and so forth. And so I think I’d entered this field very hopeful, like, I’m going to be able to figure out these important key areas where if you intervene, you’re going to be able to have a large and lasting effect, tipping points, I guess, to use, like, kind of the popular vernacular, where if you fix this thing at just this right spot, it’s going to have very long trickle out effects and, and I wasn’t finding it, and other people weren’t really finding it either. And then when we think we find one, then there’s a bunch of research trying to replicate it, and it generally doesn’t replicate.

This had just been kind of like a growing realization for me over the years, and at some point, I just thought, like, “God, this is so interesting. This is not a failure. I mean, sure what? From one perspective, it’s a failure, but, like, it’s not really a failure. This is what science is. This is how you learn, and we’ve learned something really interesting about the structure of the social world by doing these randomized control trials for so many years.” That’s where I wrote the paper. 

Edmonds: And you’ve mentioned randomized control trials a few times, RCTs, and we’ve covered them many times in this podcast, but just give me an example of how a randomized control trial might be set up in the criminal justice system in any one of the many interventions that you talked about? 

Stevenson: Sure, so maybe there’s a program within a prison, you know, it’s some sort of program that is designed to help people get on their feet when they’re released, and it entails some job training, some group therapy type stuff some maybe it extends even after you’re released and gives some subsidized employment to help you get on your feet. People, they’re excited about this program, but there’s more demand than there’s space for and so how do they decide who’s allowed in? They randomly choose people. They take a bunch of applicants, and they randomly say, “OK, well, you can come in,” and, “I’m sorry, but we don’t have space