The International School of Research Impact Assessment, Barcelona, September 15-19 / The International School of Research Impact Assessment, Barcelone, du 15 au 19 septembre

The International School of Research Impact Assessment will be held in Barcelona, Spain, on September 15-19, 2013. Kathryn Graham, a co-organizer of the five day school, shares some information about this exciting event in this guest post.

La première rencontre de « l’École internationale d’évaluation de l’impact de la recherche » a eu lieu à Barcelone, en Espagne, du 15 au 19 septembre 2013. Notre blogueuse invitée, Kathryn Graham, coorganisatrice de l’événement, nous renseigne ici sur cet atelier de cinq jours qui s’est avéré très stimulant.

There’s an increasing demand from governments and funding agencies to not only demonstrate the impact of their research investments but to optimize or get the most value out of those investments, particularly when taxpayer dollars are involved. This demand, in turn, requires skilled people to assess the impact or returns on investment.

Picture of a cartoon man scratching his head with a question mark appearing above his headOften, beleaguered research and program managers are the ones tasked to assess these impacts. But it’s a case of the demand for impact assessment outstripping the capacity for delivery. And there’s no formal school for this kind of training in the traditional academic setting.

This need was the inspiration for the creation of the first International School of Research Impact Assessment. The School will build capacity by teaching and equipping program, research and evaluation managers to deliver on the demand. It will provide the best advice, evidence and tools to assess the returns of investment, aka impact. The school is unique because it is international, practical (participants will walk away with a plan), broad in approach, high quality (roster of international experts as speakers and teachers), and a focus on impact. Although the focus will be on biomedicine, the knowledge gained will be applicable to other disciplines. Participants will come in with the needs of their own programs, which will span research activity across fields and sectors, and emerge with plans tailored to help their own organizations.

So who are we hoping will attend? All those who work in knowledge translation and program management in research and development for government, research funding organizations, academia, not-for-profits, industry or health industry.

Logo for The International School on Research Impact AssessmentAnd what can participants hope to gain? The goal of the curriculum is for participants to gain a broad knowledge of the “science of science”; develop and enhance skills for the planning and development of assessment studies, and understand how best to report and implement research impact assessments. Additionally, participants will have the opportunity to network and exchange best practices with peers from around the world.

We encourage anyone struggling or succeeding in the area of research impact assessment to apply by May 31. And for more information on how the five days will unfold, please see the Preliminary Programme.

See you in Spain!

Kathryn Graham, PhD,  Co-organizer

Jonathan Grant, PhD,  Scientific Director

Paula Adams, PhD,  Coordinating Director

The Virtual Knowledge Broker / Le courtier de connaissances virtuel

David Phipps, RIR-York

On Tuesday September 25 , I cleared my morning so that I could be the only Canadian participant in a workshop on Policy Influence and Monitoring. The workshop was in Cornwall in the UK. I was in Toronto in Canada. WebEx and Skype connected us.

Le mardi 25 septembre, j’ai libéré ma matinée afin de pouvoir être le seul participant canadien à un atelier sur l’influence des politiques et le suivi. L’atelier avait lieu à Cornwall au Royaume-Uni. J’étais à Toronto, au Canada. WebEx et Skype nous ont mis en contact.

Knowledge intermediary work is a global phenomenon. Look at the K* conference in April 2012 that was attended by participants from 5 continents. There are well established practices to enhance the impact of research on policy and practice in developing countries seeded by international organizations like the International Development Research Centre (Canada) and the RAPID program of the Oversees Development Initiative in the UK plus many more. These organizations work with local Southern partners to enhance the impact of research on the lives of citizens in the Global South.

Figure 1

I was invited by the Global Development Network (GD Net) to take part in a workshop on Policy Influence and Monitoring sponsored by the International Initiative for Impact Evaluation (3ie) who were working with a consortium lead by ODI and involving CommsConsult in UK and Zimbabwe, the Centre for Poverty Analysis (CEPA) in Sri Lanka and the Centro de Implementación de Políticas Públicas para la Equidad y el Crecimiento (CIPPEC) in Argentina. They came together to explore two questions:

  1. What is policy influence?
  2. How do we measure it?

There were about 20 participants in Cornwall plus Vanesa Weyrauch joining from Argentina, Peter da Costa joining from Kenya, Simon Batchelor based in the UK, also joining remotely and me… from Canada.

Continue reading

How to Assess the Impact of Your Research / Comment mesurer l’impact de votre recherche

Michael Johnny, RIR-York

Impact has become a significant component of the research cycle but how does one actually do this?  Dr. Sarah Morton from the University of Edinburgh offers a one-day workshop with methodologies to assess the impact of your research.

L’impact est devenu un aspect très important du cycle de la recherche, mais comment doit-on procéder pour le mesurer? La professeure Sarah Morton de l’Université d’Édimbourg offre un atelier d’une journée au cours duquel sont présentées des méthodologies permettant de mesurer l’impact de vos recherches.

David Phipps of York University (@researchimpact) has recently written about Knowledge Hypocrites.  Well, I for one am taking action (sort of).  I wouldn’t exactly call myself a bookworm, but I am taking opportunity to learn from other professionals to help inform my practice.  Perhaps not a direct solution to David’s point, but I am happy about opportunities to learn from leaders in KMb from Canada and internationally.

A recent post shared my experience attending a workshop from Peter Levesque of Knowledge Mobilization Works.  The following week I attended a day-long session led by Sarah Morton (recently Dr. Sarah Morton) of the University of Edinburgh, Co-Director Communication and Knowledge Exchange within the Centre for Research on Families and Relationships.  The topic of Sarah’s workshop was how to assess the impact of your research.

I was one of 12 present and felt I was in a unique position, whereas I am not directly involved in or leading a research project, but play a brokering role in developing research projects.  For the purposes of the workshop I used York University’s partnership with the United Way of York Region as an example (we are piloting a community knowledge broker role).  For me, in my experience, impact is misunderstood with outcomes, or even outputs.  So it was refreshing (and validating) for me to hear Sarah speak about a process of inputs, activities, uptake, use and impact.

The significant takeaway for me was a mapping exercise which will help me working with university researchers in developing knowledge mobilization plans.  In Sarah’s research and experience, embarking on a process of examining potential assumptions and risks around the process listed above can actually help determine potential indicators around impact.  Unlocking a procedure to support this process will help me in my brokering work.  The fact that I can employ the tools and not have to read Sarah’s dissertation makes my life somewhat easier.  Can I declare myself hypocrisy-free?  No, not yet, but I do prefer this active process of knowledge exchange.  My thanks to Sarah for sharing her research and methodologies to further unpack the notion of impact in research.

Rethinking Research Impact / Repenser l’impact de la recherche

By David Phipps (ResearchImpact, York)

Some new thinking from researchers helps to refine our thinking about the impact of research and how we measure the “impact” (or “contribution”) research might have on policy and practice decisions.

De nouvelles réflexions de chercheurs nous aident à redéfinir notre compréhension de l’impact de la recherche et la manière dont nous mesurons « l’impact » (ou la « contribution ») que peut avoir la recherche sur les décisions en matiere de politiques et de pratiques….

Thank you Sarah Morton. Sarah Morton is co-director of the Centre for Research on Families and Relationships at the University of Edinburgh. She came to Toronto to visit ResearchImpact-RéseauImpactRecherche (York) for 2 weeks. During that visit she made a presentation to the Southern Ontario KTE Community of Practice, met with eight civil servants from Municipal Affairs & Housing, Health & Long Term Care, Education, Food & Rural Affair and Cabinet Office and with Ben Levin’s group at Research Supporting Practice in Education (RSPE). After a weekend in Barrie’s Bay, Sarah came back to Toronto and the world of KMb for a meeting with the International Alliance of Leading Educational Institutions, York’s KMb Expo and then the KTE CoP again where Sandra Nutley (Research Unit for Research Utilization, also the University of Edinburgh) also made a presentation.

As someone from CRFR tweeted “@CRFRTweets: Sarah Morton’s met more knowledge translators in the last 2 days in Toronto than in 10 years in Scotland” (June 7, 3:45pm).

Two meetings stood out for me. Amanda Cooper presented work on evaluation of 44 knowledge intermediary organizations in education including RIR. I won’t preempt her publication by disclosing her results but her evaluation framework was made public at the Canadian Society of the Studies in Education meeting at Congress 2011 in Fredericton. She evaluated the websites of those 44 knowledge intermediaries and scored them on their presentation of their efforts for KMb products, KMb events and KMb networks. Because KMb is a people mediated process, events and networks get weighted more heavily than products in their evaluation framework. This is one of the first quantitative evaluation frameworks for a system of KMb – most frameworks measure the effects of individual KMb interventions. I look forward to Amanda’s forthcoming paper so we can have a fullsome discussion of this methodology and seek to test it in other settings. Continue reading

Michael’s ‘Aha Moment’!

I am flattered to know someone asked for a blog based on a Tweet I contributed on the ResearchImpact Twitter feed.  First, a few observations and disclaimers.  I am glad blog posts do not mirror dissertations in rigour or length.  Next, I do not claim to be an authority on ‘outcomes’ or ‘impacts’ although my work is heavily invested in both terms/processes.  Lastly, I admit I carried around strong assumptions that the logic model for impact followed a sequential (and not very quick moving) flow from activity to outcome to impact. 

January 11 and 12, I had the pleasure of attending a Scientist Knowledge Translation Training event which was hosted by The Hospital for Sick Children and was led by Drs. Melanie Barwick and Donna Lockett .  Over two days, Melanie and Donna shared practical tools for developing Knowledge Translation (KT) plans, led discussions toward a more clear understanding of KT and provided valuable exercises to improve attendees capacity to understand the ‘user context’ for successful linkage and exchange, which is a foundation for successful KT.  The 25 attendees present were predominantly health practitioners who had KT responsibilities embedded into their job descriptions although there were some health researchers and policy professionals in attendance as well.

However, back to the notion of impact.  Never one to be terribly shy, I asked about the relationship between outcomes and impact, stating my feeling it was not possible to measure impact so closely to any KT transaction because impact was a by-product of outcomes.  What triggered this question was a slide that identified short-term, intermediate and long-term outcomes.  Moreover, while this makes sense, some confusion arose for me with regard to ‘measuring impact’, which in my experience is a challenge in policy and practice-relevant research.  So when one of the facilitators commented that she would be seeking impact measures based on short-term behavioural or practice changes amongst the participants of the session, I was skeptical.  What followed was a brief discussion between us two about the relationship of outcome and impact and that it is possible to identify impact measure very closely after a KT transaction.

The ensuing discussion did not necessarily change my beliefs around impact in relation to outcomes.  Reflecting back, I would say they have expanded my beliefs.  Impact is no longer solely a longitudinal process which one must wait (pick your timeframe – 6 months, one year, five years, and so on) to identify behaviour or practice changes.

I look forward to further discussion on this topic, and the inevitable reading that I will embark upon to challenge and reinforce my expanded belief system on impact.  Given the significance of this topic for publicly funded researchers and practitioners, it is a conversation which we should all be engaged with, and a topic we should give voice to.  Hey, that could be a second ‘aha moment’!