What we Learned Around Metrics
At SciOut18 we spent a large portion of our time together discussing issues around assessment and evaluation, and worked to understand how each member of the SciOut community prioritizes the goals for these efforts. Below is a summary of our discussions.
Mechanics of Measurement: More Questions Than Answers
Understanding the efficacy of science outreach programming is critical for sustaining and scaling, yet this seems like a universal black box. It’s hard to capture our impact given the diversity of audiences and content. Furthermore, there really is no standard tool in the field for measuring effectiveness, and many of us in science outreach fields don’t have the expertise to design and/or carry out such evaluations. Yet, reporting on these metrics are essential for obtaining funding, and can help inform how changes program design/execution should be implemented to achieve better outcomes.
Emphasizing the sentiments above in her Flash Talk, Karen Kinsman raised more questions than answers when it comes to even knowing what to measure, let alone issues pertaining to how to measure. She also added the many challenges around data sharing, requirements for Institutional Review Board (IRB) approval, and the need for funders to legitimize and support the collection of “anecdata,” or the stories that we share around our experiences in the science outreach field. Yet, there does exist some structure for trying to figure it all out, at least this is the argument Sara Kobilka made in her Flash Talk around backwards design. Sara made the excellent points of learning how to prioritize what data is worth collecting, and stressed the importance of having a shared vocabulary with colleagues and collaborators as teams discuss what outcomes are most important to their context.
But even still, most practitioners in science outreach facilitate programs or events that engage outward stakeholders in a single instance. Does a singular engagement truly make a lasting impression, or do audiences have to continue showing up to experience some sort of impact on their lives? As Susan Renoe shares in her Flash Talk, the currency of evaluation involves time, money, and comfort, but if we figure out how to systematically pool the work of individual practitioners in this area, we can maximize our understanding of impact more effectively.
At least we can take solace in the idea that understanding metrics is a universal problem, and folks from a diversity of fields are working on this very issue. How can we learn what others are doing in this space? To help get another perspective outside the science outreach space we heard from Jim Liebman, a lawyer from Columbia University’s Center for Public Research and Leadership. In his Flash Talk, Jim raised the importance of establishing a theory of action, which her further dissected to include clearly defined inputs, actions, and outcomes that help us break down a theory of action into measurable steps that progress toward a desired outcome (he called this an “operationalized theory of impact”).
These flash talks provided the mental appetizer for fruitful discussions around metrics, which all attendees discussed together. More specifically, we felt that it is important to better understand the goals for science outreach, because these are the foundations on which we all build our strategies for engagement.
Goals for Science Outreach
Based on preparatory work, table level discussions, and general conversations, it is clear that there are many complex challenges around understanding the impact of science outreach. What we learned from SciOut18 attendees is that those who practice science outreach are not measuring the impact of their work, or if they are measuring impact, they are not doing it all that well. This relates to some very practical barriers such as time, funding, and general knowhow.
However, in our conversations, we did make a lot of headway learning about some agreed upon goals for science outreach, agnostic of any program or scientific discipline. According to SciOut18 attendees, the goals of science outreach are to:
1. Foster an appreciation for the process of science.
Many in the SciOut community have suggested that they wish to instill in their audiences a “wonder, excitement, and curiosity about science,” as well as honest conversations around iteration, failure, and the general pace of scientific research. However, it seems insurmountable to measure changes in how people “appreciate the process of science,” at least within teams of limited expertise in these areas.
2. Demonstrate that science is a human endeavor.
The process of science consists of a series of social interactions around the design, execution, interpretation, and communication of scientific findings. Yet, it can be difficult to demonstrate that scientific research is powered by people. Scientists have a variety of motivations, come from incredibly diverse contexts, and are subject to making (mostly) honest mistakes, yet scientists are often perceived as inaccessible and different. Because of these disconnects, it can be difficult to promote the cultivation of science identities in certain populations. Even more difficult is designing metrics that capture how non-scientist communities see those conducting science.
3. Instill trust in science and scientists.
It has been incredibly challenging to effectively translate the wealth of scientific knowledge and research findings into meaningful action for people, and there remains a significant distrust toward science and scientists (though not always generally, moreso on specific topics like vaccines and climate change, for example). Additionally, it is hard to overcome perceptions that science is not being done in the best interest of people, particularly given many grave historical injustices in the name of science. How do we work to create more trust between the scientific enterprise and non-scientific audiences? Again, this is very difficult to measure.
4. Demonstrate that science is everywhere, everyday.
Non-scientific audiences often consider science to be something that happens in expensive laboratories by elite academics. But, in reality, non-scientists are interfacing with science (and decision-making around science) on a daily basis. Science is a fundamental, though fairly silent, part of human culture. How do we think about reducing these disconnects, and event if we could reduce them, how would we know? How do we instill in non- scientists the idea that science is relevant to their everyday lives?
5. Improve access and equity with regard to scientific opportunity.
The scientific enterprise looks nothing like the world it intends to serve. This goal directly relates to improving representation in the sciences, and is the only category on this list that can be measured (mostly through participation metrics, but this is still an incomplete assessment of Diversity, Equity, and Inclusion statistics).
Because the above goals for science outreach (based on the SciOut18 meeting) tend to center on changes in human behavior, they present a great challenge to practitioners. As such, we have made the recommendation to shift our conversation on metrics to those that focus on inward stakeholders, i.e. scientists and institutions of science. We believe that understanding how culture within the scientific enterprise is changing will help inform the impact of our work in science outreach, and help drive policy that creates more inclusive spaces in science.