Measuring the Impact of Medical Research: Moving From Outputs to Outcomes

Department of Psychiatry, MGH-East, Charlestown, MA 02129, USA.
American Journal of Psychiatry (Impact Factor: 12.3). 03/2007; 164(2):206-14. DOI: 10.1176/appi.ajp.164.2.206
Source: PubMed


Billions of dollars are spent every year to support medical research, with a substantial percentage coming from charitable foundations. To justify these expenditures, some measure of the return on investment would be useful, particularly one aligned with the intended ultimate outcome of this scientific effort: the amelioration of disease. The current mode of reporting on the success of medical research is output based, with an emphasis on measurable productivity. This approach falls short in many respects and may be contributing to the well-described efficacy-effectiveness gap in clinical care. The author argues for an outcomes-based approach and describes the steps involved, using an adaptation of the logic model. A shift in focus to the outcomes of our work would provide our founders with clearer mission-central return-on-investment feedback, would make explicit the benefits of science to an increasingly skeptical public, and would serve as a compass to guide the scientific community in playing a more prominent role in reducing the efficacy-effectiveness gap. While acknowledging the enormous complexity involved with the implementation of this approach on a large scale, the author hopes that this essay will encourage some initial steps toward this aim and stimulate further discussion of this concept.

10 Reads
  • Source
    • "Academic metrics assess scholarly output rather than research impact. There is further confusion between research impact and research quality, and differentiating between terms such as outcomes, outputs and impact is challenging (Weiss, 2007). Outputs are the direct result of the project and by those directly involved in it, and include both core research activities as well as dissemination activities (Economic and Social Research Council nd). "
    [Show abstract] [Hide abstract]
    ABSTRACT: The real-world impact of research is gaining much attention across the international Higher Education sector. Funding agencies, government organisations and community groups are seeking evidence that research initiatives are delivering impact beyond contributions to academia. Researchers, practitioners, educators, learning designers and developers require a good understanding of research impact, and associated terminology, to articulate the real-world benefits of technology-enabled initiatives. There are three good reasons to understand research impact in a Higher Education context. Firstly, comprehending the language of research impact facilitates meaningful discussion with research stakeholders. Secondly, recognising and communicating the real-world impact of an initiative affirms the 'so what' factor of a research project. And thirdly, demonstrating research impact, rather than reporting research outputs, is becoming more important in funding applications and project documentation. This paper concludes with a brief review of assessment frameworks developed to evaluate the real-world impact of Higher Education research.
    Rhetoric and Reality: Critical perspectives on educational technology, ascilite 2014, Dunedin, New Zealand; 11/2014
  • Source
    • "Kuruvilla and colleagues (2006) used four main categories including research-related impacts, policy impacts, service impacts and societal impacts. Weiss (2007) criticised medical research evaluation systems as being output-based and measuring only productivity. He therefore suggested an outcome-oriented model for measuring medical research instead. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Faculty of 1000 (F1000) is a post-publishing peer review web site where experts evaluate and rate biomedical publications. F1000 reviewers also assign labels to each paper from a standard list or article types. This research examines the relationship between article types, citation counts and F1000 article factors (FFa). For this purpose, a random sample of F1000 medical articles from the years 2007 and 2008 were studied. In seven out of the nine cases, there were no significant differences between the article types in terms of citation counts and FFa scores. Nevertheless, citation counts and FFa scores were significantly different for two article types: “New finding” and “Changes clinical practice”: FFa scores value the appropriateness of medical research for clinical practice and “New finding” articles are more highly cited. It seems that highlighting key features of medical articles alongside ratings by Faculty members of F1000 could help to reveal the hidden value of some medical papers.
    Scientometrics 11/2013; 97(2). DOI:10.1007/s11192-013-0993-9 · 2.18 Impact Factor
  • Source
    • "Trochim, Kane, Graham, and Pincus (2011) 29, 30, 47, 57 22. Trochim, Marcus, Masse, Moser, and Weld (2008) 1, 3, 4, 5, 12, 22, 25, 44 23. Weiss (2007) 1, 10, 12, 21, 22, 27, 35, 36, 38, 40, 42, 47, 55, 61, 63, 69, 70, 76, 77, 78 24. Westfall, Mold, and Fagnan (2007) 20,47,54 25. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Increasing emphasis is being placed on measuring return on research investment and determining the true impacts of biomedical research for medical practice and population health. This article describes initial progress on development of a new standardized tool for identifying and measuring impacts across research sites. The Translational Research Impact Scale (TRIS) is intended to provide a systematic approach to assessing impact levels using a set of 72 impact indicators organized into three broad research impact domains and nine subdomains. A validation process was conducted with input from a panel of 31 experts in translational research, who met to define and standardize the measurement of research impacts using the TRIS. Testing was performed to estimate the reliability of the experts' ratings. The reliability was found to be high (ranging from .75 to .94) in all of the domains and most of the subdomains. A weighting process was performed assigning item weights to the individual indicators, so that composite scores can be derived.
    Evaluation &amp the Health Professions 09/2013; 37(1). DOI:10.1177/0163278713506112 · 1.91 Impact Factor
Show more

Similar Publications


10 Reads
Available from