SciOut18 Flash Talk: Karen Kinsman
https://stage.rockedu.rockefeller.edu/new_outreach/flash-talk-karen-kinsman/

Good morning! Oh come on you guys, good morning! Okay much better. I'm Karen Kinsman and for the last 16 years I've been the director of the Stem Center at the United University of New Mexico managing outreach programs educator professional development and competitions predominantly for pre-college students. My educational background is in psychology and management. I get asked all the time are you a scientist and I say no and that's really good when you run STEM competitions because nobody can ever accuse you of stepping in and overriding judges decisions because I have no idea what I'm talking about. I've been mostly involved in nonprofit management and leadership and program development for the last thirty years. For most of that time I probably like you guys have struggled with how to truly know if what we do it really has its intended impacts. We're all doing good perhaps great perhaps awesome work but how do we really know that, right? We know that intrinsically probably and we hear it but how do we really know? So spoiler alert I don't know and I haven't found the Holy Grail and I haven't met anybody who's found the Holy Grail either. So let's see if this sounds familiar and you're gonna think I'm a little crazy but close your eyes for a second imagine this: you and your team have worked tirelessly to offer outreach programs for students. Attendance is great, maybe sometimes not-so-great. Students seem to be genuinely engaged and they're post-event surveys bear that out the students and maybe even families or teachers if your audience is pre-college are over overheard talking about how awesome the event was how much they learned and perhaps even that it changed how they see their own futures. Their survey answers seem to indicate that your event really made them think positively about STEM careers. They can see themselves as a scientist or an engineer or a programmer or a physician. You're also fairly certain that a number of your participants are also participating in similar programs offered by other organizations doing outreach in your area. You always wonder which program is having more impact. It must be yours right? Or if participating in more than one experience over time is what actually creates lasting impact. Okay you can open your eyes now I won't make you close them for the rest of the time. Any of that sound familiar to anybody? vaguely? Okay so here's one of the first big questions that came up for me and doing the pre-work: how can we best measure the effectiveness and impact of our programs to me to funders to our partners to our stakeholders? I think that's the million-dollar question sometimes quite literally the million dollar question or some of us hope it's a million it may be more like a thousand so I'm gonna disappoint you one more time by admitting I don't have that answer. Either I have discussed this very topic what seems hundreds of times in the past 16 years with a collection of cross sector partners we have searched for answers and it seems we mostly just keep adding to our list of questions. Here are just a few of our other unanswered ones see if any of them ring true for you How do we really know that what we're doing has any long-term lasting impact? How do we know if the biggest impact comes from one program one event or participating in multiple programs over time? How do we even approach the idea of defining impact? How might that definition change given different contexts? Let me throwout a couple more words you might recognize and I'm betting that some of you are gonna cringe. Anecdotal data oh and thank you Jeannie for my new vocabulary word anecdata. I love that! so anecdata is my new vocabulary word for our organization and I would venture to guess many of yours as well anecdotal data is a very informative and powerful metric for us as practitioners but is not necessarily highly valued or lacks credibility amongst funders and related agencies because it is not considered hard data, right? It's not numbers necessarily. How do we go about working with funders and other agencies to legitimize the very real anecdata that we receive from our participants? I know another unanswered question. Let me throw something else at you and this was another kind of theme in my pre-work was data sharing. Any of you ever talk about data sharing with colleagues? What's your reaction to those two words? Mine was oh god that's a big hairy beast. That big hairy beast brings with it even more questions and unfortunately not a whole lot of solid answers at least in where I exist so things like how do we make it feasible to share data across organizations or institutions when we serve the same or similar populations? This would allow us to perhaps track participants who engage in multiple programs to help us determine if or how that participation impacts for instance their decision to pursue a STEM career but we seem to make it extraordinarily difficult to actually share information even and I heard this conversation at my breakfast table and I hear it at my work all the time even at our own institutions we cannot figure out how to share information you know god forbid we share information and figure out that we could actually develop collective impact and increase how we impact other people but we can't do that because you might take my funding or you might take my idea or I might take yours which is not the point and I come from the nonprofit world outside of academia until 16 years ago and that's totally not how it works in the nonprofit world for the most part. For the most part we're all about getting together and figuring out how to share stuff because there's limited stuff out there but for whatever reason we can't figure that out in academia and I'm not there's my other caveat I'm not an academic either which is probably why I don't understand that. If we could share data what data would be worth sharing? What data would actually inform our practices and make the most impact? Are there feasible ways to share participant data across organizations and institutions? Here's a big question that's come up in our cross sector partnership discussions who would own the data? All right here's another one is IRB approval required and if yes see you're laughing too and if so who's IRB will be consulted and will IRBs at other partner institutions accept the decision of another institutions committee? I dare you to repeat that. That's been a pretty recent conversation in our our cross sector partnership group how and where would the data be secured what data would actually illuminate trends for us that would really make a difference and how do we engage with alumni of our programs after they've graduated and gone into higher education or into the workforce. I have a counterpart at UNM who actually works on the undergraduate side of the world we have no way of knowing if any of the students that he's working with as freshmen and sophomores participated in my programs and whether or not their participation in our programs before they went to college impacted their interest in stem got it almost done so the list goes on and on as a community of practitioners and stem outreach I think we have yet to be able to truly answer those questions. Well I don't know about you but when we seem to get close to answering some of the questions we just come up with more that need to be answered and I cannot imagine we're alone so I have a single example about sharing anecdotes of how we figure out if students are participating in our programs with other organizations I have one stellar student who's been with us since she was in sixth grade and I just found out by happenstance that she's participating in another organization's programs and is actually teaching younger students science but I wouldn't know that except for I was sitting at a table in another context all together with the educator from that organization who said oh I think you might know Michaela. Huh yeah I do I took her to the International Science Fair twice so how do we get beyond our questions? Can we develop solutions that are useful across a variety of outreach programs? And can we find our way past obstacles real and perceived that keep us from truly being able to take advantage of the amazing opportunities that could exist to create collective impact that could really move the needle for our participants? I think we're here to try to start finding solutions to at least a few of our shared unanswered questions and I'm so excited to see what we can come up with together so let's do it! you