Measuring the Impact of Medical Research: Moving From Outputs to Outcomes

Department of Psychiatry, MGH-East, Charlestown, MA 02129, USA.
American Journal of Psychiatry (Impact Factor: 13.56). 03/2007; 164(2):206-14. DOI: 10.1176/appi.ajp.164.2.206
Source: PubMed

ABSTRACT Billions of dollars are spent every year to support medical research, with a substantial percentage coming from charitable foundations. To justify these expenditures, some measure of the return on investment would be useful, particularly one aligned with the intended ultimate outcome of this scientific effort: the amelioration of disease. The current mode of reporting on the success of medical research is output based, with an emphasis on measurable productivity. This approach falls short in many respects and may be contributing to the well-described efficacy-effectiveness gap in clinical care. The author argues for an outcomes-based approach and describes the steps involved, using an adaptation of the logic model. A shift in focus to the outcomes of our work would provide our founders with clearer mission-central return-on-investment feedback, would make explicit the benefits of science to an increasingly skeptical public, and would serve as a compass to guide the scientific community in playing a more prominent role in reducing the efficacy-effectiveness gap. While acknowledging the enormous complexity involved with the implementation of this approach on a large scale, the author hopes that this essay will encourage some initial steps toward this aim and stimulate further discussion of this concept.

  • Source
    • "Academic metrics assess scholarly output rather than research impact. There is further confusion between research impact and research quality, and differentiating between terms such as outcomes, outputs and impact is challenging (Weiss, 2007). Outputs are the direct result of the project and by those directly involved in it, and include both core research activities as well as dissemination activities (Economic and Social Research Council nd). "
    [Show abstract] [Hide abstract]
    ABSTRACT: The real-world impact of research is gaining much attention across the international Higher Education sector. Funding agencies, government organisations and community groups are seeking evidence that research initiatives are delivering impact beyond contributions to academia. Researchers, practitioners, educators, learning designers and developers require a good understanding of research impact, and associated terminology, to articulate the real-world benefits of technology-enabled initiatives. There are three good reasons to understand research impact in a Higher Education context. Firstly, comprehending the language of research impact facilitates meaningful discussion with research stakeholders. Secondly, recognising and communicating the real-world impact of an initiative affirms the 'so what' factor of a research project. And thirdly, demonstrating research impact, rather than reporting research outputs, is becoming more important in funding applications and project documentation. This paper concludes with a brief review of assessment frameworks developed to evaluate the real-world impact of Higher Education research.
    Rhetoric and Reality: Critical perspectives on educational technology, ascilite 2014, Dunedin, New Zealand; 11/2014
  • Source
    • "Kuruvilla and colleagues (2006) used four main categories including research-related impacts, policy impacts, service impacts and societal impacts. Weiss (2007) criticised medical research evaluation systems as being output-based and measuring only productivity. He therefore suggested an outcome-oriented model for measuring medical research instead. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Faculty of 1000 (F1000) is a post-publishing peer review web site where experts evaluate and rate biomedical publications. F1000 reviewers also assign labels to each paper from a standard list or article types. This research examines the relationship between article types, citation counts and F1000 article factors (FFa). For this purpose, a random sample of F1000 medical articles from the years 2007 and 2008 were studied. In seven out of the nine cases, there were no significant differences between the article types in terms of citation counts and FFa scores. Nevertheless, citation counts and FFa scores were significantly different for two article types: “New finding” and “Changes clinical practice”: FFa scores value the appropriateness of medical research for clinical practice and “New finding” articles are more highly cited. It seems that highlighting key features of medical articles alongside ratings by Faculty members of F1000 could help to reveal the hidden value of some medical papers.
    Scientometrics 11/2013; 97(2). DOI:10.1007/s11192-013-0993-9 · 2.27 Impact Factor
  • Source
    • "The specific aim of the study is to assess the level of implementation of results from a subgroup of research projects (dealing with respiratory diseases), based on the opinion of researchers and a different typology of healthcare decision-makers as well as to identify the barriers or facilitators of translating new knowledge into informed decision-making. The study also addresses the multidimensional issue commented by Weiss [19] on the decision-making process that is differentiating awareness from implementation and the similarly important aspect of who takes the role of decision-maker or person in charge of translating research outputs into daily practice (implementation). "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background This article reports on the impact assessment experience of a funding program of non-commercial clinical and health services research. The aim was to assess the level of implementation of results from a subgroup of research projects (on respiratory diseases), and to detect barriers (or facilitators) in the translation of new knowledge to informed decision-making. Methods A qualitative study was performed. The sample consisted of six projects on respiratory diseases funded by the Agency for Health Quality and Assessment of Catalonia between 1996 and 2004. Semi-structured interviews to key informants including researchers and healthcare decision-makers were carried out. Interviews were recorded, transcribed verbatim and analysed on an individual (key informant) and group (project) basis. In addition, the differences between achieved and expected impacts were described. Results Twenty-three semi-structured interviews were conducted. Most participants indicated changes in health services or clinical practice had resulted from research. The channels used to transfer new knowledge were mainly conventional ones, but also in less explicit ways, such as with the involvement of local scientific societies, or via debates and discussions with colleagues and local leaders. The barriers and facilitators identified were mostly organizational (in research management, and clinical and healthcare practice), although there were also some related to the nature of the research as well as personal factors. Both the expected and achieved impacts enabled the identification of the gaps between what is expected and what is truly achieved. Conclusions In this study and according to key informants, the impact of these research projects on decision-making can be direct (the application of a finding or innovation) or indirect, contributing to a more complex change in clinical practice and healthcare organization, both having other contextual factors. The channels used to transfer this new knowledge to clinical practice are complex. Local scientific societies and the relationships between researchers and decision-makers can play a very important role. Specifically, the relationships between managers and research teams and the mutual knowledge of their activity have shown to be effective in applying research funding to practice and decision-making. Finally the facilitating factors and barriers identified by the respondents are closely related to the idiosyncrasy of the human relations between the different stakeholders involved.
    Health Research Policy and Systems 05/2013; 11(1):15. DOI:10.1186/1478-4505-11-15 · 1.86 Impact Factor
Show more