Quantitative Evaluation (and a little shameless self-promotion) / Évaluation quantitative (et un peu d’autopromotion éhontée)

By David Phipps, RIR-York

Amanda Cooper (@ACooperKMb) recently released her evaluation of 44 Canadian Research Brokering Organizations. She presents a quantitative method for evaluating the effort of a system of knowledge mobilization.

Amanda Cooper (@ACooperKMb) a récemment dévoilé son évaluation de 44 organisations canadiennes de courtage de recherche. Elle présente une méthode d’évaluation quantitative visant à mesurer les efforts d’un système de mobilisation des connaissances.

Knowledge mobilization struggles with evaluation.  Evaluating an individual instance of knowledge mobilization is feasible with the right base line and pre/post intervention metrics. But rolling that up and evaluating a system of knowledge mobilization (like any one of the knowledge mobilization units in the ResearchImpact-RéseauImpactRecherche network) has so far proven challenging.

So thank you, Amanda Cooper (Assistant Professor in the Faculty of Education at Queen’s University, Kingston, Ontario). Amanda recently posted a report titled “Knowledge mobilization in education: A cross-case analysis of 44 research brokering organizations across Canada”. Amanda developed a quantitative methodology to evaluate the efforts of Canadian research brokering organizations (RBOs). The methodology is based on the evidence about research utilization. We know that people centred methods encourage greater research use than do those based solely on making package knowledge accessible to decision makers. In the words of Sandra Nutley and her colleagues in Using Evidence, “[p]ersonal contact is crucial … studies suggest that it is face-to-face interactions that are most likely to encourage policy and practice uses of research” (page 74).  In Amanda’s methodology points are assigned depending on how the RBO employs products (12 points), events (20 points) and networks (20 points) as well as overall features (20 points). You can see that more points are assigned to people centred methods (events and networks) than are assigned to purely product based methods. How points are assigned is detailed in Appendix B of her report.

Amanda used RBO’s web sites as the data source and scored each of the 44 RBOs on a scale out of 100. Amanda cites ResearchImpact as one of the RBOs but the data she used pulled from York’s Knowledge Mobilization Unit. The Harris Centre, another RIR member, is also included separately as one of the 44 RBOs.

Key Point #1: This is a quantitative methodology that is reliable and reproducible citing satisfaction with the inter-rater reliability testing of the tool and the average intra-class correlation coefficient.

Key Point #2: This method evaluates a system of knowledge mobilization not the efficacy of an individual knowledge mobilization intervention.

Key Point #3: This method measures the efforts of Canadian RBOs. It does not measure impact of the RBOs efforts. That more effective RBO efforts will result in greater impact of those efforts is a testable hypothesis, but it makes sense that this would be the case.

Key Point #4 (shameless self-promotion alert): RIR-York achieved the highest score in this study.

Each with a score of 81%, RIR-York tied with the Fraser Institute and Canadian Education Association as the top performing RBOs. Fraser Institute achieved this score with a budget of $12.8M. CEA achieved this score with a budget of $2M. York’s budget for knowledge mobilization is approximately $250,000. RIR-York accomplished the same effort on a fraction of the budget. The data from the top nine ranked RBOs is presented below.

Rank

 

Organization

 

 

 

Type*

 

 

 

Size (FTE)

Operating

Expenditures

Score on KMb Matrix (%)

1

1.2.1 RI

NfP, university research centre

 

 

Small (3)

$250 000

81

1.2.4 Fraser

NfP, think tank

 

Large (60)

$12,808,690

81

1.4.2 CEA

Memb, network

 

Small (9)

$2,044,892

81

2

1.2.4 AIMS

NfP, think tank

 

Small (5)

$872 234

78

3

1.2.0 CCL

NfP, general

 

Large (77)

$20,583,490

76

1.2.3 The Centre

NfP, issue-based

 

Large (25)

$5,685,000

76

4

1.2.0 TLP

NfP, general

 

Large (74)

$5,293,039

75

1.2.1 HC

NfP university research centre

 

 

Med (11)

75

5

1.2.0 CCBR

NfP, general

 

Med (12)

74

6

1.1.2 E-BEST

Gov, district level

 

Small (6.5)

72

7

1.2.1 CEECD

NfP, university research centre

 

 

Small (9)

69

1.2.2 P4E

NfP, advocacy

 

Small (9)

$537,806

69

1.2.3 LEARN

NfP, issue-based

 

Large (33)

$3,000,000

69

8

1.2.1 HELP

NfP, university research centre

 

 

Large (50)

$7,200,200

67

9

1.1.3 CSC

Gov, standards

 

Large (20)

$3,849,254

65

We need more research like this into the processes of knowledge mobilization, engaged scholarship and community based research. Much of what we know comes from individual studies of individual instances of knowledge mobilization. As these activities become more embedded in institutions and systems we will increasingly need research on these systems and how they create infrastructure to support the individual instances. You can read more on other methods for evaluating the impact of research like Payback and Productive Interactions in a 2011 Special Edition (Volume 20, Number 3) of the journal, Research Evaluation.

Thank you to Amanda for your important contributions to this emerging field.

The Virtual Knowledge Broker / Le courtier de connaissances virtuel

David Phipps, RIR-York

On Tuesday September 25 , I cleared my morning so that I could be the only Canadian participant in a workshop on Policy Influence and Monitoring. The workshop was in Cornwall in the UK. I was in Toronto in Canada. WebEx and Skype connected us.

Le mardi 25 septembre, j’ai libéré ma matinée afin de pouvoir être le seul participant canadien à un atelier sur l’influence des politiques et le suivi. L’atelier avait lieu à Cornwall au Royaume-Uni. J’étais à Toronto, au Canada. WebEx et Skype nous ont mis en contact.

Knowledge intermediary work is a global phenomenon. Look at the K* conference in April 2012 that was attended by participants from 5 continents. There are well established practices to enhance the impact of research on policy and practice in developing countries seeded by international organizations like the International Development Research Centre (Canada) and the RAPID program of the Oversees Development Initiative in the UK plus many more. These organizations work with local Southern partners to enhance the impact of research on the lives of citizens in the Global South.

Figure 1

I was invited by the Global Development Network (GD Net) to take part in a workshop on Policy Influence and Monitoring sponsored by the International Initiative for Impact Evaluation (3ie) who were working with a consortium lead by ODI and involving CommsConsult in UK and Zimbabwe, the Centre for Poverty Analysis (CEPA) in Sri Lanka and the Centro de Implementación de Políticas Públicas para la Equidad y el Crecimiento (CIPPEC) in Argentina. They came together to explore two questions:

  1. What is policy influence?
  2. How do we measure it?

There were about 20 participants in Cornwall plus Vanesa Weyrauch joining from Argentina, Peter da Costa joining from Kenya, Simon Batchelor based in the UK, also joining remotely and me… from Canada.

Continue reading

How to Assess the Impact of Your Research / Comment mesurer l’impact de votre recherche

Michael Johnny, RIR-York

Impact has become a significant component of the research cycle but how does one actually do this?  Dr. Sarah Morton from the University of Edinburgh offers a one-day workshop with methodologies to assess the impact of your research.

L’impact est devenu un aspect très important du cycle de la recherche, mais comment doit-on procéder pour le mesurer? La professeure Sarah Morton de l’Université d’Édimbourg offre un atelier d’une journée au cours duquel sont présentées des méthodologies permettant de mesurer l’impact de vos recherches.

David Phipps of York University (@researchimpact) has recently written about Knowledge Hypocrites.  Well, I for one am taking action (sort of).  I wouldn’t exactly call myself a bookworm, but I am taking opportunity to learn from other professionals to help inform my practice.  Perhaps not a direct solution to David’s point, but I am happy about opportunities to learn from leaders in KMb from Canada and internationally.

A recent post shared my experience attending a workshop from Peter Levesque of Knowledge Mobilization Works.  The following week I attended a day-long session led by Sarah Morton (recently Dr. Sarah Morton) of the University of Edinburgh, Co-Director Communication and Knowledge Exchange within the Centre for Research on Families and Relationships.  The topic of Sarah’s workshop was how to assess the impact of your research.

I was one of 12 present and felt I was in a unique position, whereas I am not directly involved in or leading a research project, but play a brokering role in developing research projects.  For the purposes of the workshop I used York University’s partnership with the United Way of York Region as an example (we are piloting a community knowledge broker role).  For me, in my experience, impact is misunderstood with outcomes, or even outputs.  So it was refreshing (and validating) for me to hear Sarah speak about a process of inputs, activities, uptake, use and impact.

The significant takeaway for me was a mapping exercise which will help me working with university researchers in developing knowledge mobilization plans.  In Sarah’s research and experience, embarking on a process of examining potential assumptions and risks around the process listed above can actually help determine potential indicators around impact.  Unlocking a procedure to support this process will help me in my brokering work.  The fact that I can employ the tools and not have to read Sarah’s dissertation makes my life somewhat easier.  Can I declare myself hypocrisy-free?  No, not yet, but I do prefer this active process of knowledge exchange.  My thanks to Sarah for sharing her research and methodologies to further unpack the notion of impact in research.

Looking forward to Social Innovation / Dans l’attente de l’innovation sociale

By David Phipps (ResearchImpact York)

What is social innovation? If knowledge mobilization (the process) results in social innovation (the outcome), knowledge brokers need to capture and evaluate social innovation outcomes that arise from knowledge mobilization.

Qu’est-ce que l’innovation sociale? Si la mobilisation des connaissances (le processus) conduit à l’innovation sociale (le résultat), les courtiers de connaissances doivent alors identifier et évaluer les résultats de l’innovation sociale qui découlent de la mobilisation des connaissances.

I have recently read four interesting and thought provoking pieces on social innovation.

The Power of Social Innovation (2010) by Stephen Goldsmith presents the lessons learned from a two term mayor of Indianapolis including case studies from education, poverty and housing. This book is also backed up by a website and the author may be found tweeting as @powerofsocinnov.

The Philanthropist (2010) released an Open Access edition (Volume 23, Number 3) of their journal featuring articles about Canadian social innovations. The entire edition and individual articles can be found here. This journal featured articles written by Stephen Huddart of the McConnell Foundation and Frances Westley of SIG @ Waterloo among others.

On February 6, 2011, The Huffington Post published the interview Rahim Kanani had with Judith Rodin, President of the Rockefeller Foundation, on the Evolution and Promise of Social Innovation. There are wonderful stories about social innovation, collaborations and some lessons learned by the Rockefeller Foundation.

“Horizons” is published by Canada’s Policy Research Initiative. In February 2011, Horizons published a number of articles on social innovation and the community sector. This follows their earlier briefing on social innovation that did little more than ask the questions. You can read their analysis and their answers online here by clicking on the left hand menu bar. There are many articles including ones on social impact bonds, public sector innovation and an interview with Louise Pulford of the Young Foundation in the UK.

All of these complement the national work of the Canadian Task Force on Social Finance who tweet as @socialfinance and the international reach of the Ashoka Foundation who tweet as @ashokatweets.

But what is it? What is Social Innovation?

In The Philanthropist, Geraldine Cahill published “Primer on Social Innovation: A Compendium of Definitions Developed by Organizations Around the World”. This includes definitions of social innovation drawn from practice around the world. In 2009, ResearchImpact-York published the following definition:

Social innovation is “the creation or application of research and knowledge to develop sustainable solutions to social, environmental and cultural challenges. Social innovation results in more efficient and effective human services, more responsive public policies and greater cultural understanding.”

It’s a good definition as it links to results but the most elegant definition I read was from Louise Pulford in which she defines social innovation as “devising new and better ways to tackle social problems”. This is simple and easily understood but what relationship does it have to knowledge mobilization (KMb) and the work of ResearchImpact-RéseauImpactRecherche and the many engaged scholarship and research projects underway in Canadian communities and universities?

In our 2009 paper we linked our definition of social innovation (above) to KMb by saying that “KMb (the how) enables social innovation (the what)”. We have recently refined this to say that social innovation is the outcome of the process of knowledge mobilization. KMb isn’t an end in itself. KMb is a process that allows researchers and their partners to develop social innovations which are new and better ways to tackle social problems. At RI-York we have spent 4 years focusing our attention on the process of KMb but without looking we have also supported some examples of social innovations:

  • The Parkdale Activity and Recreation Centre (PARC) Heat Registry. The Heat Registry is a community based innovation that tracks and provides services to poor and vulnerable populations at risk of heat exposure on hot summer days. The Heat Registry is based in part on the research and evidence collected by Tanya Gulliver, a York University Knowledge Mobilization Intern whose research helped the PARC Heat Registry raise operating funds that benefited the community and citizens of Parkdale. The increased risk of heat exposure due to poverty is an example of a social determinant of health.
  • In the summer of 2010, York U and United Way of York Region jointly supported three interns who undertook research to map social assets in York Region. The work of these three interns previously appeared in Mobilize This! on June 21, 2010 and their work provided data and evidence to inform a UWYR Board decision to launch Strength Investments, a new form of UWYR investment that invests in the strengths of coalitions of York region citizens and community groups.
  • Susan Lloyd Swail (former MES graduate working with Professor Gerda Wekerle) was a KMb intern jointly funded by Nottawasaga Futures and MITACS Accelerate program. Her research contributed to the development and launch of the South Simcoe Green Economy to Nottawasaga Futures (see Mobilize This! from April 13, 2010) . Susan is now Manager of their Green Economy Centre that provides knowledge and services to local business to improve and maximize the efficiency of Green Business Practices.

As we grow in our KMb efforts we need to evaluate on outcomes (social innovation) as much as on process (KMb). We first wrote about social innovation in our 2009 paper. Now that more people are writing about and acting on social innovation we will also update our thinking and re-energize our actions to capture and evaluate social innovation outcomes as well as KMb processes.

ResearchImpact-RéseauImpactRecherche is still Canada’s knowledge mobilization network but now we know that we need to articulate those social innovations that emerge from our KMb efforts.

Thirty four mobilizers walk into a bar…

CocktailsSSHRC invited 34 knowledge mobilization projects from their Knowledge Impact in Society and SSHRC Clusters to a workshop in Ottawa October 22-23.

Day 1: Not being challenged by systemic introversion our mob of mobilizers (mostly academic leaders, some project coordinators and two lone staff leading institutional knowledge mobilization services at David YetmanYork and Memorial) had no problem mashing up in different combinations be it in their KIS or Cluster cohort or the sector of primary engagement. Most of the day was spent exploring “issues” around knowledge mobilization. The usual topics of incentives, barriers, metrics & evaluation were on the agenda. Refreshingly some new topics including an alleged research/KM dichotomy and social media were also discussed.

Research vs KM got a lot of play with opinions on both sides of and in between the hypotheses that research and KM are either on a spectrum of activities or they are two sides of a coin, related but separate. ResearchImpact works with researchers, their institutions and their non academic research collaborators to create Clair Donovanspace for basic research AND space for applied research linking to extra academic impact (thank you Clair Donovan) as well as a spectrum of activities and services in between. KM is a process intimately interwoven with research. It is not a discrete event that happens in isolation of the research. Measures of extra academic impact complement, they do not conflict, with measures of academic quality. A repeated theme was the desire for infrastructure (cash, expertise, systems) to support the spectrum between basic research and extra academic impact.

twitterAlso interesting was a breakout session on social media. ResearchImpact tweeted @researchimpact during this session resulting in a number of RTs and DMs – on the spot web 2.0 mobilization of knowledge about knowledge mobilization. SSHRC, our academic researchers and their non-academic research collaborators only need to look at Surfertheir graduate students to see how social media will play an increasingly important role. You don’t have to lead the wave but if you don’t ride it, it will pass you by.

York is definitely leading the wave. With a total of 5 engaged research and knowledge mobilization projects York has by far the best representation of any Canadian university at this meeting. ResearchImpact was pleased to be joined by Canadian Homeless Research Network, Canadian Refugee Research Network, Canadian Business Ethics Research Network and the Toronto Employment Immigrant Data Initiative.

Tiedi, CBERN and Homeless Hub

BeerReception done. Dinner done. Blog written. Beer being consumed thanks to Southern Cross Grill. Need to recharge before day 2 of this important workshop. Thank you SSHRC for creating this space where 34 mobilizers could walk into a bar and begin to network. Trouble was it was a cash bar and alcohol is not an eligible expense on a SSHRC grant even though it is a key success factor in networking and knowledge mobilization! Maybe we’ll make that a recommendation for future program development.

ResearchImpact Evaluation Survey

The KM Units which are leading the ResearchImpact network (York University and the University of Victoria) are undergoing an evaluation, and we would like your feedback!  We are asking you to complete a survey regarding your experiences with the Knowledge Mobilization Unit. The purpose of the survey is to gather information  on how a specific research partnership may have influences you, your organization, and/or your community. The survey should take approximately 15 minutes of your time and will be greatly beneficial in the improvement of the KM Unit. Please access the survey by clicking this link.