What is your data trying to tell you? In this episode, John Dues talks to Andrew Stotz about why most leaders misread data, overreact to single results, and miss the real story. Discover how Deming thinking exposes when change is truly happening and how to use a process behavior chart to listen to the real story. Plus, find out why nine years of 'stable' results may still demand transformation. Tune in and rethink data-driven leadership! 

 

0:00:02 .2 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we dive deeper into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is understanding variation is the key to data analysis. John, take it away.

 

0:00:27 .8 John Dues: Andrew, it's good to be back. Yeah. So, we've just started the school year in Ohio, so I thought doing a session on goal setting would be a good place to kick off the year. And I was thinking a lot of leaders, school leaders and leaders in general, are setting goals around this time period. And I was really thinking about having this Deming lens. I was thinking, how did I set goals before I sort of started understanding this approach? And it's, you know, this is one of those things where if you really stop and think about it, goal setting is a lot harder than it seems at first glance. Things like, how do you set a reasonable goal? And then once you've gotten to that place, how do you know if things are improving? How do you know if things are getting worse? And I was thinking how powerful this understanding variation method is for folks that may be struggling with those questions.

 

0:01:32 .9 Andrew Stotz: Yeah. In fact, that's a great question for the listener and the viewer. Like, how do you set goals? How did you set goals in the past? How have you improved that? And I was thinking when you were speaking, I didn't set goals. I gave proclamations. You know, 20% of I want to see this and that. And they were just stretch targets without any means or methods. So yeah, interesting.

 

0:01:55 .2 John Dues: Yeah. How do you set the target? Was it arbitrary? Is it based on some standard that you heard somewhere? A lot of times you have no idea sort of what's behind that target or you've sort of associated it to something that's familiar. Like in my case, we often sort of set goals that sort of mimic the grade scale. So, you know, 80% is a common goal for something like test scores, you know.

 

0:02:23 .7 Andrew Stotz: But they don't even call them goals anymore. They call them, let me remember, I think it's called KPIs.

 

0:02:30 .0 John Dues: KPIs, targets, you know, lots of different things for sure. And I think what I've seen is that a lot of the reason that goal setting is so hard is because you, well, one, you misinterpret your data in the first place. And a lot of that misinterpretation, at least in the education sector, is because leaders don't have the knowledge. They don't know about natural variation. They're typically making comparisons between some current performance level, some previous value. But those two things, those two data points don't show you, don't convey the behavior of that data across time. So, what we do and what I did before I sort of discovered this method is you overreact to a single data point. Probably less frequently, you underreact to the data because you don't have this understanding of, you know, how much is the data moving up and down sort of naturally almost no matter what you're doing. Now, that's not always the case, but that's the case that I've found in a lot of situations. And so until you start to take that into account, those natural ups and downs, then you just misinterpret the data over and over again, usually by overreacting is what I've seen.

 

0:03:54 .9 Andrew Stotz: Yeah.

 

0:03:56 .0 John Dues: So there's, you know, I think as a starting point, people in the Deming community will be familiar with, a lot of people. But others listening to this probably have never heard of this idea of dividing variation into, I've heard it described as like two flavors. There's the routine variation, what I call natural variation, things vary naturally no matter what you're doing. And then there's exceptional variation where things are so different that there is reason to pay attention to this. And what I found through studying this is, the key is knowing how to tell the difference between those two types of variation. And don't do that, lots of confusion, lots of wasted effort. And so that's really where the power of this methodology comes into play. And for anybody that's studied this, you sort of realize that you have to have a tool to make that differentiation. It's not arbitrary. And so that's where what I call the process behavior chart, some people call the control chart, where that comes into play because that tool allows us to tell what type of variation is present. And it also allows us to tell if the system is predictable or unpredictable. And once we have that understanding, then we can chart an improvement sort of roadmap that makes sense.

 

0:05:21 .8 Andrew Stotz: Yeah. In fact, I've applied two of the things, you know, one of the things to my pass rates and admission rates, I applied the process chart, the control chart, based upon your recommendation a long time ago. And it did help me to kind of think if, you know, in my case, I wanted to break out of that standard outcome that I was getting. And so I realized, something has to change substantially in the system in order to get a different result than this variation that I was getting. That was the first thing. And then the second one, a couple of nights ago, I was giving a lecture and I was using your work that you and I have talked about, as well as Mike Rother's stuff on goal setting and having the target. And then there's that obstacle. And what I realized when I gave a little talk on it and I used the diagram and I showed the obstacle, it became kind of apparent to everybody like, oh, yeah, there's an obstacle there that we don't know how to solve.

 

0:06:27 .6 John Dues: Yeah.

 

0:06:28 .3 Andrew Stotz: And that's where PDSA came in. And we started talking about that, as you have taught previously. So, yeah, I'm excited to hear what you have to say today.

 

0:06:38 .2 John Dues: Yeah. And the Mike Rother model, I mean, he does have this target that's this long term target that's pretty hard to hit. And you don't really know what you're going to do. But the difference there then in the situation I'm describing is that that in Mike's model, that target is knowingly outside of the current sort of capability of the system. And they're coming together as a team and saying, how do we get to that target six months from now or a year from now? And we're working towards that rather than someone has just arbitrarily set some target, without a realization that the system isn't capable of hitting that currently. Those are two completely different scenarios. Yeah. So, I think I'll share my screen. Well, actually, before I do that, I would just say, too, because I know sometimes when I introduce these things, a lot of times people get scared away because they think the math is hard. And what I would say there is that there's the creation of a process behavior chart probably takes about fourth grade level math skills. You really only need to do addition, subtraction, multiplication, and division.

 

0:07:49 .3 John Dues: That's it. But the thinking, I think, actually can be taught all the way down to the kindergarten level. And I've actually seen kindergartners explain the data on a process behavior chart. So, if anybody gets scared away at this part, the math is simple and the thinking is also pretty simple and powerful once you sort of have the basics. So, I'll go ahead and share my screen so the folks that are watching have a visual to follow along on. And for those that don't, I'll do my best to describe it. When we're talking about a process behavior chart, and this one's sort of an annotated version so that things are clear. But basically a process behavior chart is just a time sequence chart. It has upper and lower natural process limits, and we plot data for some measure that we're interested in. And the chart typically has a central line so that we can detect a trend of those plotted values toward perhaps either limit. So, this particular chart, the data is the percent of students who scored proficient or higher on the Ohio third grade reading state tests from spring 2004 through 2015. 

 

0:09:06 .8 John Dues: So, I've labeled sort of some of those key parts of the chart. So, just kind of call those out. Again, the red lines a