Photo from Evaluating Social Innovation Prototypes: A Guide. The fourth step in the evaluation process consists of three tasks: Review the Findings, Judge the Merit of the prototype, and Make a Decision about the future of the prototype.

By Mark Cabaj

“Evaluating Social Innovation Prototypes: A Guide” is a practical resource for those involved with social research and development and who would like to create, test, and learn from prototypes. This how-to guide explores 12 principles to guide the testing process, five key steps for carrying out the process, and includes tables that summarize a variety of prototyping techniques, evaluation methods, and sampling strategies. Learn how to effectively use prototypes and test promising solutions to address social challenges.

Why were you personally interested to author this guide?

So many people in the field – including me – have talked about the importance of using social innovation prototypes as a way to engage more people in surfacing, developing and testing promising ideas to make progress on stubborn societal challenges. Yet, while everyone seems to agree on ‘why’ we need to evaluate prototypes that can be generated through this process, there is actually not a lot of material out there on ‘how’ we should go about it. Producing this guide with SI Canada and the score of people who contributed to it is a modest contribution.

Who can benefit from the Prototype Evaluation Guide?

Anyone interested in using social innovation prototypes as part of their efforts to make progress on societal challenges. This includes those who facilitate social change processes, the participants in those processes, and the funders, partners and decision-makers who may support them. There is a little bit of something for everyone in this guide.

How has your research into creating this guide changed how you prototype?

I’ve realized that while improvisation is a necessary feature of social innovation and social change work, it does not mean that it is ok to ‘wing it’ when it comes to evaluation. While we always have to be creative in how we evaluate and learn from our work, it’s easier to do that when we are guided by some simple structure, principles and common practices. The Prototype Evaluation Guide tries to provide some of this.

Of the prototyping you’ve seen being done recently, which of the principles (mentioned on page 16) are most often overlooked? Why do you think that is?

Two stand out. The first is remembering that the primary purpose of testing a prototype is to NOT assess its impact, it’s to learn: (1) about the potential impact of promising idea, (2) if and how the idea might be improved, and (3) whether the idea has enough merit to warrant making a ‘bigger bet’ in terms of a pilot project, adopting it or even scaling it. The work of assessing impact comes later if and when an idea being prototype is actually implemented in a real-life setting.

The second is the importance of being smart in who we engage to get feedback on our prototypes, otherwise known as ‘sampling’ in evaluation research. The most common sampling strategy is simply to reach out to people we know and are easy to connect with. This is called convenience sampling. It’s easy, but not very robust. I think we need to employ many other more helpful strategies laid out in the Guide if we want to take prototyping seriously; for example, engaging with those people who represent different ‘stakeholders’ of the promising solution, or even who we suspect might be quite critical of the idea, which can challenge our own blinders about the limits of what we are proposing.

One of the principles for prototyping is to be culturally responsive. How do you recommend prototype teams try to mitigate the unequal power relationships involved in research?

There is a lot of good thinking and practice emerging in this area, so I encourage people to explore the many perspectives out there on this important topic. I will share three things that always come to my mind first using a real-life example of an initiative where a group wanted to test various ways to better alert migrant workers – a group experiencing unequal power for multiple reasons – in a U.S. state on their ‘rights’ to COVID-19 related health services during the pandemic. Here is what they did. 

First, they ensured that migrant workers had a meaningful role in shaping the evaluation questions used to test a prototype, were involved in reviewing the feedback on the prototypes and participated in making decisions about next steps for different prototypes.  

Second, where and when possible, migrant workers helped to identify and employ the techniques used to gather and make sense of the feedback on a prototype, particularly when it came to gathering feedback from their peers. For example, the group decided against using a survey administered by a consultant when migrant workers reported that one-on-one interviews, carried out by migrant workers with migrant workers on the job site, would result in much greater participation rates and more honest feedback than a survey sent by phone or email from someone they did not know or trust.

Finally, they made it a point to ensure that when the evaluation findings were shared the voices of migrant workers were clearly visible, rather than ‘buried’ in some general statements about what ‘stakeholders’ said. They learned that while community health center administrators and public health managers strongly preferred an idea for a larger scale education campaign because they felt it could reach the most people, most migrant workers felt that it would not work well for reasons that the prototype team – and health administrators – never anticipated. 

Three starting point ideas. Easy to describe. Harder to do. Yet necessary and worth it because they contributed in a small way to helping create a more equitable power relationship in the prototype evaluation process. 

There is an appendix of resources at the end of this guide for more information. Is there a particular person, organization, and/or project involved with social innovation and prototyping that you are finding to be especially inspiring lately, and you can tell us more about it?

I have always followed Nesta the social innovation foundation in the U.K., because their team is so good at documenting their work and they pay close attention to how to take ideas often used in the private sector and adapt them for use in the ‘social’ sector. I consider them a bellwether organization whose work I need to stay abreast of. In the process of writing the Prototype Evaluation Guide, I became aware of the good work done by the United Nations Development Program (UNDP) in Asia, particularly their thinking on how to work with – and evaluate – ‘portfolios’ of prototypes. This has really stretched my thinking on this and made me realize our social innovation field still has a lot of work ahead of it to reach its full potential.