How do we really know when improvement has happened inside a school or organization? In this episode, John Dues and Andrew Stotz unpack a clear, three-part definition of improvement and show why evidence, method, and sustained results matter far more than year-to-year comparisons. Their discussion offers a practical lens for leaders who want to distinguish true progress from noise and build changes that last.

TRANSCRIPT

0:00:02 .4 Andrew Stotz: My name is Andrew Stotz and I'll be your host as we dive deeper into the teachings of Dr. W. Edwards Deming. Today I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is, How To Define Improvement. John, take it away.

 

0:00:23 .3 John Dues: Hey, Andrew. It's good to be back. Yeah, I think this is really interesting. Apologies on the front end. I'm a little bit under the weather, so I may sound a little raspy today. But you know, one of the things that's really interesting is there's lots of claims of improvement. In my world, there's lots of claims of school im- improvement. I would even go as far as to say that those claims are like a dime a dozen, something like that. And the reason I say that is not to be mean or anything, but you know, I think that a lot of these claims, they're not grounded any kind of reasonable evidence. And I think sort of even beyond that, that claims are often made without a logical definition of improvement. So I thought in this episode we could talk about a three-part definition that makes it really easy to tell when improvement has occurred and just as importantly, when it hasn't.

 

0:01:21 .9 Andrew Stotz: Exciting.

 

0:01:23 .2 John Dues: Yeah. When I talk about this, I always like to start with a challenge. So, you know, if I'm in a workshop, I'll say, you know, get out a piece of paper and a pen so the listeners could do this as well and think about, you know, the successful improvement efforts that you've led throughout your career. So in my world, maybe it's increase in state test scores or maybe you improved student enrollment in your school. Maybe you did a better job at retaining the teachers in your school. It could be any number of things. Maybe it's decreasing student office referrals or decreasing chronic absenteeism rates in your school or your school system, which are two things on everybody's mind coming out of the pandemic especially. And I tell people, just create a list of those instances. And I give them a few minutes usually. And typically, people come up with eight, nine, 10 or so instances of improvement, whether that's teacher in their own classroom or principal in their school, or a superintendent thinking about the whole system. Then I say to them, now what I want you to do is pause and think, what does it mean to improve?

 

0:02:46 .7 John Dues: What do you mean by that? And that really brings us to this important question. What is improvement? You know, and this was... Full disclosure, when I started thinking about this, I stumbled across the definition in a book I'll show you here in a second. But when I stumbled across this, you know, there was some conviction. And I think that probably a lot of educational leaders or just, you know, leaders in general would say, actually, I never really thought about that. I don't have an answer for this seemingly simple question. And like I said, I didn't have an answer to that question when I really thought about it, when I stumbled across the definition, probably for the majority of my career, maybe the first 20 years or so, if I'm at year 25. So, yeah, the first two decades, I would not have had a clear answer for that simple question. Now, I turn to this seminal work in the field of improvement science called The Improvement Guide. I'm sure a lot of people are familiar with this. And I'll share my screen so people can see the book and kind of share an interesting story about the book. And, you know, when you're... Can you see my screen all right now?

 

0:04:06 .9 Andrew Stotz: Yep.

 

0:04:08 .1 John Dues: So you can see, if you're just listening, you can see the covers of two books. So on the left, a lot of people will recognize The Improvement Guide. But there's an arrow up there. It says, second edition. And a lot of people will recognize that book. Probably less people. Maybe some people that have been doing improvement work for maybe three decades will know this other book, the first edition of The Improvement Guide. It's this purple book on the right, if you're watching. But there's this interesting anecdote that I actually think I might have heard maybe on your podcast when some of the authors were on. And almost as soon as they wrote this first edition, this purple edition, they got this note from this professor in Brazil, and it said, I know you guys are really big into improvement, and you're really big on operational definitions, but you've written this whole book on improvement, and nowhere in the book have you defined what you mean by improvement. So, you know, talk about a swing and a miss.

 

0:05:15 .0 Andrew Stotz: Yeah. And just for listeners out there, you can go to the podcast.deming.org and you can search for quality as an organizational strategy with Cliff Norman and Dave Williams. We didn't talk about The Improvement Guide specifically, but definitely it's worth listening to those two.

 

0:05:33 .3 John Dues: Yeah, and I think... So any of us that feel bad when we come to realize, like, how obvious that question is when we've made claims of improvement and don't have a definition. So even these guys that were writing a whole guide about how to do improvement missed as well. You know, and it's pretty obvious when you think about it that The Improvement Guide, that a book like that should have a clear definition for the central concept. That's right in the title. But it should be just as obvious to leaders that they also need a definition of improvement. And that definition should really precede any improvement claims then. So I thought it'd be interesting to take a look at the definition that the authors came up with in their improved second edition.

 

0:06:20 .9 Andrew Stotz: Improvement in their improvement.

 

0:06:24 .4 John Dues: Improvement in their improvement guide. Right. And the definition is really easy to follow. It's got three parts, and now I've adopted it into my own improvement work. But what they've said is improvement is "a change that alters how work is done or the makeup of a tool that produces visible positive differences relative to historical norms and relevant measures and it's sustained into the future." So we can kind of break that three parts of the definition down now. So in part one of the definition, what you have to be able to do is point to a change that was made that led to better results. You know, that could be a new tool you're using, a new approach, a new framework, maybe it's a new staff role, but something has to change, a new method, something has to change in what you're doing. So that's sort of part one of the definition. Part two is performance improved after the change compared to past results. So that also should be fairly obvious. So you did something different. You noted when you started this new thing, and at some point in relative proximity to when you tried that new thing, the data improved.

 

0:07:54 .2 John Dues: You know, it went up. If that's the direction of good or, you know, if it's like chronic absenteeism or office referrals, you want it to go down. But you see that in the data after you've made this change, that's part two. And then part three is that improvement after the change was sustained into the future. So it wasn't just a temporary thing because you were paying a lot of attention to it, but you made the change. The data improved over time after you did this new thing, and then it kept going into the future.

 

0:08:27 .9 Andrew Stotz: Which is the hardest part, by the way.

 

0:08:30 .1 John Dues: The hardest part, I'd say too. Assuming you could bring about improvement, then sustaining it into the future, especially as you maybe take your eye off it a little bit, and then work on something else, that's very, very difficult.

 

0:08:43 .1 Andrew Stotz: Yeah, the initial seizure that you get into of making change can be really powerful compared to the energy, you know, devoted to sustaining it.

 

0:08:55 .9 John Dues: Yeah.

 

0:08:56 .3 Andrew Stotz: And you could also argue, if somet