SciOut18 Flash Talk: Jim Liebman
https://rockedu.rockefeller.edu/new_outreach/flash-talk-jim-liebman/

Okay, so I'm Jim Liebman I'm Columbia Law professor believe it or not I run a Center at Columbia and what we do is we bring upper level graduate students from all over the country to spend a semester with us learning how to be improvement engineers in whatever career they choose to go into but we Really? Hope they will go into and try to get them into the public sector, especially focused on public education and what we do is to teach them how to do measurement for improvement and one of the ways we teach them is to get them to work with us in interdisciplinary teams doing Performance measurement projects all over the country and in Brazil, so if you think you might need our help, let me know So what we try to teach them and we teach them this in 70 hours of instruction Which is going to be compressed into 15 minutes here. So I've given you this PowerPoint and my power Slide notes so you can follow up on a lot of the detail here that I won't be able to discuss with you but basically the idea here is in a sense to not have to have the evaluator because you build into Everything you're doing every day the opportunity as Jeanne said to learn from failure. And so we call that evolutionary learning building into everyday activity Measures of your success and so a little story very quickly the United States in the early 1950s Realized that if it could build a nuclear submarine one that did not have an internal combustion engine in it It could be under the water for a long long period of time and essentially change the strategic balance across the world but there was a problem which was it's very dangerous to put a nuclear reactor inside of a sub and a lot of Sailors in there with it and there was no baseline. Nobody knew how to do this. Nobody Nobody had ever done it before and it wasn't the kind of thing that you'd say well Let's go do it put the men in the submarine ship them out there and so yet a few accidents will learn from that. So Admiral Hyman rickover had to figure out how you could build something like this which to this day In the nuclear Navy in the United States. There's never been a reactor accident there have been Russians and the Japanese Phelps, but not ours and the way he figured out to do this is basically This evolutionary learning process and that was to set expectations Really clearly for exactly what you were going to do with every little bit of the design and engineering And building process so big issue Wells What were they going to weld the casing for that nuclear reactor with we're gonna use silver they were going to use cadmium So what they did was they said for every single weld we are going to predict exactly how much pressure that weld can hold and then they would do the weld they would try the pressure on it and they would compare the outcomes and if the outcome was Different in any way if the outcome was better than they thought or worse than they thought that was an error They did not have full information and so they then did a problem-solving process to figure out where they'd made their mistake in the Expectations and then they come up with a new Expectation and they would do another weld and they would try those out until they got every one of their welds coming out exactly as they Predicted so that is evolutionary learning or learning from mistakes learning from error. Okay so by the way Toyota beat the American automakers and the European ones using the same method you'll have some materials like that in your Conference so they're really three steps here Which are be clear about what you expect to do and to happen in order for you to succeed measure whether your actual actions and results match your expectations if you find deviations between Expectations and results. That's great. You've got something you can learn from and so And and the big thing to do I'm gonna give you lots of ideas here don't take them All the basic idea is will they help you learn and improve if you don't think so Skip them or customize them too what you might want to do. So, how do you get clear about what you expect to do? And happen in order to succeed. So what we recommend is a very simple idea Come up with a theory of action. Remember that first word. It's a theory it's a hypothesis It's just what you think you're gonna do in order to get where you want to go But it's not anything more than that And the only way to test it is by putting it into effect and then you test your theory. So a theory of action Typically works in very simple if then as a result statements if we do this we expect these things to follow and after that we expect our impact or our desired aim to follow so here is an example of a theory of action that we came up with for and school system Imaginary school system named appleson and they're trying to create a program to help middle school maths Students be ready for what they need to accomplish when they get to high school So here's the theory of action for that and I'm not going to read it But a theory of action then is a first step but it's not enough. It's way too High level. So what we have come up with is this idea of an operationalized theory of action which breaks down your theory of action what you expect to do into a set of inputs that will support actions, which will lead to Outcomes outcomes are what you expect to see in changed beliefs changed behaviors Changed intermediate results that will then lead to your impact So you translate your if your if statements into those inputs and actions you translate your then? Into outcomes and your as a result into impact and then what you do so here for the Appleton example is the Operationalize theory of action. So some inputs, etc. All there. You can take a look at that Okay. So now you have made clear to yourself what you expect to do and To happen and that's really important. You always think well, we know what we want to do But if you try to lay it out with some clarity You'll learn a lot about it and you'll find out you really didn't know everything you thought you did. So it's helpful It's a strategic tool even if you're not measuring, okay? Now what you want to do is measure Whether you get the results that you thought you were going to get and it's not just the results Did you put the inputs in place to do take the actions? And when I say take the actions did you do the things you wanted to do at the level of quality that you wanted to? do them do they have those intermediate results did people start changing their mindsets changing their behaviors and things like that and So that's why we're going to measure There are lots of different kinds of measures there process measures those are the things that measure whether you did what you thought you were going to do and you did it at the level of Quality that you wanted to do, they're leading indicators. They're really important if they're going wrong forget about the outcomes You don't have to wait for the outcomes You're likely not to get to the outcomes because you're not doing what you expected to do or not doing it very well. So this gives you a lot of early warning about what to do. Then you can measure your outcomes. These are those intermediate things that you expect to see happening changes in beliefs and behaviors. Then There are these other kinds of measures that are talked about there. So here is an alignment of the Appleton operationalized theory of action with some indicators of each of crucial things that they were going to do as inputs, as actions and we come up with an indicator of what will tell us that we did what we thought we were going to do at the level of quality we thought we wanted and getting these outcomes that we thought we would get and then an indicator for the ultimate impact that you are seeking. And so Lots of questions to ask about measures. Are you going to do process Or both? Measures, quality, and don't forget about quality We often want to measure how many people came to the training but we don't measure whether they learned anything. So that's an important thing to do. I'll talk a little bit more about tools. We think it's really important for you to set a target for each one of those indicators where you expect to get as a number or as some kind of a quality that you can measure So that you have that expectation set that you can compare to your actual results and if you find a discrepancy, you can learn from all of that. So here are a bunch of kinds of tools, surveys, in-person protocols, administrative records, you collect all sorts of data just because you write down the names of everybody who says they're going to come or things like that. That's administrative data. That's there that you collect in any event standardized assessments and rubrics are another which is a rubric isn't something you use when you're doing an observation. And so In setting your targets something you want to do is benchmark So you want to benchmark against your own past performance or against the performance of somebody else who's doing something similar. And these are questions that you want to ask. You don't have to have a measure for every input action, etc Pick the ones that are important. Pick the ones that you think you can measure best and these are some of the questions you can ask about whether to come up with a measure and what kind of measure and all of that. So ability to gauge quality and quantity feasibility it's best to use data that you already have even if it's kind of a proxy for what you're looking for. Will it reveal variation? It's important and you want to be able to embed it in your everyday operations. So you're doing your own evaluation. You don't need that evaluator to spend all that money on and then there is this process of analyzing deviations between what you thought we were gonna do or what's gonna happen and what actually happens. How much time do I have? Four minutes. Okay, so I'm just gonna go through this really quickly. The problem-solving process actually is the most important one. Sometimes we get really excited about measuring and we've got all those measurements and then It looks like things aren't going as well as we thought and that's where we stopped. No, that's actually The most important point is to come and this is it. This is a Problem-solving method that we have come up with others use some very similar to this So you've got to organize your effort get your team together that you want to use to figure out what's going wrong And the comments that that earlier talk about bringing people together who are actually the consumers of it. They're really good people to have on the teams. Analyze the problem to do that you want to figure out cause you're not going to do this deductively. You're gonna come up with some practical methods for guessing about what we're learning about causes. Then you want to refine your problem and your aim. This is actually really important in thinking about that ultimate impact. We have some tools for how you come up with what your ultimate aim is. Then you develop a strategy for solving the problem and the really most important thing to do is to come up with quick ways to test your strategy and by quick I mean an hour Or two days or at most a couple of weeks where you try to pick some part of that strategy out run it in front Of some kind of an audience and then see if it is working because why waste the time coming up with a big solution implementing it across all your scale if it's not a great idea. And then the rest that we have I have here here's some thinking about how you come with that problem statement. Here's how you come up with thinking about what your aim is. Smart goals just like you've heard before. And this is some problem, some causal analysis: the five whys for coming up with possible causes, the fishbone exercises for coming up with possible causes. Here's one filled out for an example that's traced through. All of these things that I've just mentioned. And then when you want to come up with a solution you say okay, what's our aim? What kinds of big changes do we think we need to have happen to get to that aim? What actions would we have to take to make those changes occur? And this is a driver diagram of that sort filled out for the example that I've been writing through here. And then there's this test cycle that you can go through. So the main thing here is the one thing I said, we also talked about prototyping here, which is just Do things that are practical and you think will help you improve learn and improve. Sorry, I had to go through it so quickly but there is a process to this. There are a number of organizations that help others as we do to build this into your everyday activities. My question is where do we find these slides? I'm going to send them on the slack. I'll be putting them on the slack and then if you need to me to email them or share them I can do that too. That was the best question possible.