Simulation: a strategy for success in quality and safety in pay-for-performance environments.
ABSTRACT Pay-for-Performance appears to be another step in our ever-changing healthcare environment. In most of the white papers, reports, and web pages devoted to improving the quality of healthcare in America, there is a failure to recognize or list medical simulation as a methodology to reduce the costs of implementation and to speed transition to the new order. The Agency for Healthcare Research and Quality is funding research in simulation to improve quality. This article outlines the rationales for using simulation and how simulation can benefit all involved. With a paucity of proof that simulation can deliver in terms of improving the quality of healthcare, the mass of evidence has been from observation and anecdotal tales of medical professionals that simulation is a valid tool. This article correlates the use of simulation in other nonmedical pay-for-performance professions to similar situations in medicine as some other evidence that simulation should be considered a viable option. I conclude by relating the individual strengths of simulation to the six quality initiatives of the Institute of Medicine's second report from the Committee on Quality of Health Care in America. Simulation can work to enhance the assimilation of change with each of these initiatives and help to reduce the costs of doing so. There are limitations to simulation, but used within those limitations, simulation should prove to be a powerful tool.
- [Show abstract] [Hide abstract]
ABSTRACT: Validity is critical for meaningful assessment of surgical competency. According to the Standards for Educational and Psychological Testing, validation involves the integration of data from well-defined classifications of evidence. In the authoritative framework, data from all classifications support construct validity claims. The two aims of this study were to develop a categorization method for validity evidence published in support of surgery performance assessments and to summarize the results of applying this methodology to the gynecologic surgery literature. This was a critical analysis of published observations reported as validity evidence in studies with a construct validity claim. Medline and Embase databases were searched using keywords: "surgery" and "construct validity". Parameters included English-language articles published from 2000 to 2012. Gynecologic studies were analyzed for definitions of construct validity and nonstandard terminology. Categorization criteria were developed and applied by the researchers to all observations. Two independent evaluators examined reported observations for compliance with guidelines provided by the Standards. Inter-rater agreement was calculated using weighted kappa. The initial search returned 167 articles. Twenty-five articles were left for inclusion in our analysis. Eighteen (72 %) articles defined construct validity as the ability to discriminate between expert and novice levels of proficiency. Within the sample, 80 discrete observations of reported validity evidence were identified and categorized according to standard classifications. Nearly 30 % of all published observations intended to demonstrate differences in performance by level of proficiency, 25 % described a scoring model, and 14 % demonstrated support of assessment content. Not one article contained a statistical correlation between assessment scores and objective outcomes from the authentic surgical environment. Medians for level of rigor ranged from 0 to 1 across all forms of evidence. Weighted kappa values ranged 0.60-0.91. Validity claims in gynecologic surgical assessment over-rely on generalizability evidence. No test-criterion evidence was observed. Increased awareness of current standards and systematic argument development is needed for gynecologic performance assessments.Advances in Health Sciences Education 11/2014; DOI:10.1007/s10459-014-9548-y