Working PaperPDF Available

Research impact assessment

  • June 2017
DOI:10.13140/RG.2.2.18850.43205
Authors:
Samantha Battams at University of Adelaide
Samantha Battams
Download full-text PDF
Read full-text
Download citation

Figures

Figures - uploaded by Samantha Battams
Author content
All figure content in this area was uploaded by Samantha Battams
Content may be subject to copyright.
Content uploaded by Samantha Battams
Author content
All content in this area was uploaded by Samantha Battams on Jun 14, 2017
Content may be subject to copyright.
HEALTH OUTCOMES INTERNATIONAL
5A Glynburn Road, Glynde, SA 5070 | P: 08 8363 3699 | F: 08 8365 3560
E: info@hoi.com.au | www.hoi.com.au
RESEARCH IMPACT ASSESSMENT
A Health Outcomes International White Paper
DR SAMANTHA BATTAMS
DATE 9TH JUNE 2017
HOI white paper
9th June 2017
| 1
RESEARCH IMPACT ASSESSMENT
1.1 DEFI N I N G TERMS
When measuring research impact, it is important to define terms and distinguish between stages
involved in the research process. Research impact refers to evidence for the application of knowledge
making a difference, e.g. to health status, quality of life or cost effectiveness. Knowledge translation
essentially refers to the review, synthesis, assessment and application of knowledge by key
stakeholders to improve health status or quality of life (Graham et al., 2006).
1
Research impact is linked to research utilisation, the process for implementing ideas. This can be
distinguished from knowledge transfer which refers to collecting and sharing ideas, knowledge and
research results to enable new products/services. Knowledge exchange also involves ‘collaborative
problem solving between researchers and decision-makers.
2
Figure 1: A staged approach to identifying impact of selected research projects
Step 2: Knowledge transfer and
exchange
What processes were in place for
knowledge transfer and exchange
for the projects? How effective were
these processes?
Step 1: Research purpose and
outputs
What substantive results was the
research expected to yield?
What were the underlying
assumptions behind the research
objectives?
What were the research outputs?
Step 4: Research impact
assessment and sustainability
What evidence is there for the
application of knowledge making a
difference to key stakeholders
(service users/population groups)?
Were there unintended
consequences of the research? How
sustainable are research outcomes?
Are processes in place to ensure the
spread of beneficial research
outcomes?
Step 3: Knowledge translation
What evidence is there for the
application of knowledge?
1
Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J., Caswell, W., Robinson, N. (2006). Lost in knowledge translation:
Time for a map? The Journal of continuing education in the health professionals, 26, 1, 13-24.
2
Canadian Foundation for Healthcare Improvement, http://www.cfhi-
fcass.ca/PublicationsAndResources/ResourcesAndTools/GlossaryKnowledgeExchange.aspx
Considering the concept of ‘project spread’ may be
particularly useful in complex research environments
involving multiple organisations and funders.
HOI white paper
9th June 2017
| 2
1.2 KNOWL E D G E TO ACT I O N PROCESS
To arrive at ‘research impact’, a ‘knowledge to action’ process should ideally be in place. Such a cycle
is analogous to ‘reflexive practice’. Stages of Graham’s (2006) ‘knowledge to action’ cycle include:
1) Problem identification and the review of appropriate knowledge to address the problem.
2) Adapting knowledge to local contexts, assessing barriers to knowledge and tailoring and
implementing interventions.
3) Monitoring knowledge use (collecting evidence on who used the knowledge and how, e.g. policy
change or guideline development).
4) Evaluating outcomes (the impact of using the knowledge).
5) Sustaining knowledge use (applying the stages to ongoing use of the knowledge).
An action research cycle (involving an ongoing cycle of planning, acting, observing, and critically
reflecting) would align to this process. Monitoring and reflecting upon knowledge use and research
utilisation (the process for implementation of ideas) should be an important part of this action
research cycle.
1.3 MEASU R I N G PROJEC T SPREAD
The measurement of ‘project spread’ is an important consideration when considering research impact.
We note that the term ‘spread’ is drawn from the Institute for Healthcare Improvement (IHI)
Framework (2006).
3
. A key requirement identified within this framework is to assess whether
improvements have been spread within or across the selected projects from the perspective of:
• Leadership, which incorporates executive leaders ensuring that the ‘spread’ aim is aligned with the
broader organisational direction and assign day to day responsibility for the spread of processes
to a team.
• Setup for spread, which relates to developing the infrastructure to support broader adoption of
change, including:
o Engagement of key figures or partners (internal and external) to support and promote
participation in and adoption of changes
o Identification of successful sites from which to draw support, input and learning
o Identification of those likely to readily adopt changes
o Determination of the population to be targeted; and establishment of a plan for attracting
those likely to readily adopt the changes
3
Massoud MR, Nielsen GA, Nolan K, Schall MW, Sevin C. A Framework for Spread: From Local Improvements to System-Wide
Change. IHI Innovation Series white paper. Cambridge, MA: Institute for Healthcare Improvement; 2006
http://www.ihi.org/resources/pages/ihiwhitepapers/aframeworkforspreadwhitepaper.aspx
HOI white paper
9th June 2017
| 3
• Marketing change, relates to the development of a case for adopting the proposed changes and
to clear describe the changes (or better ideas to use IHI terminology)
• Communication, ensuring that those responsible for leading spread and identified key individuals
use appropriate communication to promote awareness of the changes or improvements
• Social system, providing an environment that supports adoption of change through provision of
appropriate technical support, identification and response to barriers and developing and using
individuals to continue promotion of the changes
• Knowledge management, collecting and responding to information about the spread of change
or improvement
• Measurement and feedback, collecting and reporting process and outcome indicators to
highlight achievements of the changes and monitor progress in spreading the change within or
across organisations.
• Utilisation of this framework implies a series of both process and outcome measures.
1.4 A PROGRA M LOGIC APP R O A C H
A program logic can act as a broad guide when evaluating a program or project. It entails
consideration of the various steps taken toward achieving an outcome, and the relationship between
those steps. It establishes a causal pathway between the various steps and long-term outcomes (the
theory of change for the project). A program logic model is developed by first considering the long-
term outcome to be achieved by the project, and then working backwards to determine the short and
medium-term outcomes that will achieve this, along with the various inputs, activities and outputs
required. The result is known as an ‘outcomes hierarchy.’ There are several tools for developing
program logic models, including the Victorian government’s Evaluation Toolbox and Flinders
University’s Planning and Evaluation Wizard.
4
Figure 2: Program Logic Model
Evaluation Toolbox, Victorian Department of Environment and Sustainability
5
4
http://som.flinders.edu.au/FUSA/SACHRU/PEW/dac_focus.htm
5
http://evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=30&Itemid=136
HOI white paper
9th June 2017
| 4
1.5 MULT I P L E ORGANISAT I O N S AND FUN D E R S
In reality, research is often undertaken within complex organisational environments with multiple
stakeholders, partners and project funders. Considering the concept of ‘project spread’ may be
particularly useful in complex research environments involving multiple organisations and funders.
Additional factors which may be considered when developing research and implementing research
project results across organisations environments are:
• Joint development of research problems and a common understanding of research context
(joint definition of ‘problems’ to be addressed by research, and potential strategies for addressing
them)
• Acknowledging tension between research and organisational goals and incentives (e.g.
academic professional incentives) and seeking potential ways to address them (e.g. through
processes for defining research problems and negotiation)
• Organisational culture (e.g. academic professional culture, culture within large organisations)
which may be an important consideration in terms of research translation.
• Leadership across levels within an organisation to implement research findings (strategic to
operational)
• Cross organisational structures for implementation of research findings
• Project champions who may be particularly important within some projects (e.g. clinical
champions or leaders, or organisations working with clinicians on practice change)
• Considering current structures and systems across organisations, and how they may have a
role in monitoring knowledge use and translation
Kotter’s 8 stage process for leading organisational change
6
may be useful when trying to implement
research results and achieve research impact within complex organisational environments. This entails:
• creating a sense of urgency (the need for change)
• building a guiding coalition to lead the change
• forming a strategic vision and initiatives
• enlisting ‘a volunteer army’ to support change
• enabling action by removing identified barriers
• generating short term wins
• sustaining the change, and
• institutionalising the change.
6
https://www.kotterinternational.com/8-steps-process-for-leading-change/
HOI white paper
9th June 2017
| 5
More information?
Health Outcomes International (HOI) is a
specialist health care and community
services consulting firm. The HOI team has
extensive experience in consulting and
management in the acute, subacute, primary
and community-based services, mental
health, disability services and Aboriginal
community controlled sectors.
Contacts:
Lilian Lazarevic
Managing Director, HOI lilian@hoi.com.au
Dr Samantha Battams
Associate Director, HOI samantha@hoi.com.au
Web: hoi.com.au
Telephone: (08) 8363 3699
Post: PO Box 2065, Glynde
Plaza LPO, Glynde, SA 5070
Location: 5A Glynburn
Road, Glynde, SA, 5070
ResearchGate has not been able to resolve any citations for this publication.
ResearchGate has not been able to resolve any references for this publication.