SciOut18 Flash Talk: Susan Renoe
https://rockedu.rockefeller.edu/new_outreach/flash-talk-susan-renoe/

I am not totally a memorizer but if I don't have some kind of notes we'll be here much longer than seven minutes and nobody wants that. My name is Susan Renoe and I am the Assistant Vice Chancellor for Research, Extension, and Engagement. If you want to know what that title means there's probably someplace we can meet later and talk about it but I'm also the PI for the National Alliance for Profit Impacts. We're happy to be sponsoring this event, we're really excited about it. You've heard people say, the last two speakers say, I don't have any answers right because I wasn't asked to give the answer that's why I'm standing here. Jeannie had said we want you to figure out evaluation metrics and come and talk about it for seven minutes I said no thank you. In fact when I got the homework and I saw that we had to talk about evaluation I think how many feel like that sometimes I thought that all the time I go to meetings and conference and we talk about it and I'm like oh this is awful right so I'm going to talk to you about it from four perspectives. So I'm gonna talk to talk to you from the perspective of Assistant Vice Chancellor for Research, Extension, and Engagement whose job it is to measure the impact of the outreach engagement that we're doing across our entire campus. Can you hear me? Now okay I'm also gonna talk to you from this perspective of being a PI. I'm also going to talk to you from the perspective of someone who works with researchers all the time to work on their broader impacts but all other kinds of outreach activities and plans. I'm also going to talk to you from the perspective of someone who runs outreach programs. Now you see one when I think about evaluation my brain wants to explode. Right so we talked for a minute about how do we measure impact collectively at an institution. How many of you have read the NSF Pat G? Now when they talk when program officers talk to researchers about that they're like you read that if you want to but it's a good idea if you're bored you want to go to sleep read it. I read it all the time because there's lots of little tidbits about broader impacts and things that are in there right so I'm going to make sure that I get everything that I know what I'm doing and there's a little bit of information about evaluation. How many of you know that researchers are supposed to include evaluation in their runner and text plans? Right but if you read a little bit further it says the National Science Board recognizes that one researcher doing evaluation is good and interesting but there's value in aggregating data. How many people on your campus are all using the same evaluation scheme? I see no hands right so we have to figure out how to get get sure who doesn't do evaluation at all none see look he shaking his head no Jory who does a survey, Laurie who does a focus group, Kay who does another kind of survey, but it's not the same questions as Jory right so there's no way to put that together so that's a problem. Right how do you know how big the average NSF budget is for a regular person? It's about a hundred thousand dollars a year so let's do some simple math because this is one of the things I do with my researchers and they're like oh my god Here's a hundred thousand dollars a year right then there's a little pesky thing called what indirect right and indirect can be anywhere from 33% to like what seems like a hundred percent right but what is that please use 50 right so hundred thousand dollars shrink that in half right. Now everyone writes in a graduate student right that's half your money. Now you want to put in some sort of summer salary for yourself. Then you actually want to do some research and you might have to buy some sort of equipment and all of a sudden Susan says you have to do money for your broader impacts right because broader impacts aren't free right you actually have to do pay the people who are doing your outreach and engagement. Right, now the hands are stuck together and then Susan says what? You have to write any money for who an evaluator that's when the wheels fall off the bus right because then I ask the researchers how many people have you know how many of you know an evaluator? Now we are not the right group to ask that right but our researcher our researchers are like um I don't know right or if they do know somebody I'll say how many of you met that evaluator because you knew somebody who knew that evaluator. Right I have to say it I'm one of those people that when I wrote maybe my first grant I picked the evaluator that somebody I knew recommended right. I didn't necessarily go find the right evaluator for the right project right so one of the things that we do we think about ease if I'm not really good at evaluation which also is costly right because it costs money so a lot of times my evaluator costs more money than the outreach program I'm actually evaluating. right choices I have two minutes Thank You Jory So the problem is for researchers and for all of us there's time in doing evaluations there's cost to doing evaluation and there's a comfort level. How many of you flew through LaGuardia when you came? How many of you saw the happy buttons? I walked into the bathroom and there's an evaluation thing I can push the button happy man hop in you sit in the line for the taxi queue there was another one. How are you happy with the service in the taxi queue? Now a couple of things that were interesting one is they didn't ask me if I was unhappy with my experience in the bathroom because they know there's only four stalls in the women's bathroom I was really unhappy but the question wasn't that the question was how is the cleanliness of the bathroom? It was clean right so that's something else. First thing about how do we what is it that we are measuring the other thing that comes out and somebody mentioned a little bit earlier talking about impacts when I go and I talk to at NSF this is no joke I don't talk about broader impacts I talk about broken impacts and why do I say that? One to get people's attention but two because we don't you asked the question. How do we know if something's been impactful? Karen said something that how do we know whether it's the cumulative effect of many things that make people go on and do wonderful things in science or something else right? My daughter my daughter has 15 different careers she wants right now she wants to be an attorney a corporate attorney and a marine biologist studying sharks. See the irony there right? So true in any case my daughter goes to lots of stem activities. Right she went to a stem robotics camp probably three years ago. No one has followed up to find out if she likes science any better because she went to that robotics camp. No one's even asked if her grades and science are better because she went to that and I guarantee in another six years no one is gonna be asking whether A she went to the University of Missouri for school because she went to that robotics camp or B if she's studying thank you three minutes He's got more time than you George he's tagged I got three minutes left just saying maybe they had a reset because of the whole microphone but no one's also checking to see if she's gonna become a marine biologist because she had a great experience in that stem camp or anything to do with that. Right that's longitudinal data that we should be tracking to figure out the actual impact because otherwise our broader impacts activities are just activities that we think are good to do and we think are gonna be good for people right. What's the problem with longitudinal tracking? it's hard to find them. What else? It's expensive and it takes time. Right the only people on my campus I know who do longitudinal tracking are people who are making programs that are getting kids into graduate school and the professorial and their funding depends on them doing that right and what's the scientific way that they do that? Facebook. They get on Facebook and they figure out what they're doing right and they update that a couple of them also finds try to figure out where they aren't saying with them and send them postcards right so here's the deal: Evaluation is important. We know that the new grant that we just wrote the evaluation budget is the largest chunk of money we're spending in that why because we have to know if what we're doing matters and if it's important and I guarantee you that when I go ask for more money they're gonna ask me what did you do with the money we gave you right? So cost, ease, comfort, understanding. What to do and the ability to look longitudinally and aggregate things together are the things that I'll make my face goal when it comes to evaluation. So my hope is that over the next day and a half we're going to talk about that. We're gonna think about how we can do that systematically. One of the things we're doing on our campuses is we're trying to figure out a system for helping our researchers do that better and I'm happy to talk about that some more so thank you for your time. I have lots of questions no answers and I'm looking forward to the rest of the next two days!