Meaningful Variation in Performance: A Systematic Literature Review

Division of Research, Kaiser Permanente Medical Care Program, Oakland, CA, USA.
Medical care (Impact Factor: 3.23). 02/2010; 48(2):140-8. DOI: 10.1097/MLR.0b013e3181bd4dc3
Source: PubMed


Recommendations for directing quality improvement initiatives at particular levels (eg, patients, physicians, provider groups) have been made on the basis of empirical components of variance analyses of performance.
To review the literature on use of multilevel analyses of variability in quality.
Systematic literature review of English-language articles (n = 39) examining variability and reliability of performance measures in Medline using PubMed (1949-November 2008).
Variation was most commonly assessed at facility (eg, hospital, medical center) (n = 19) and physician (n = 18) levels; most articles reported variability as the proportion of total variation attributable to given levels (n = 22). Proportions of variability explained by aggregated levels were generally low (eg, <19% for physicians), and numerous authors concluded that the proportion of variability at a specific level did not justify targeting quality interventions to that level. Few articles based their recommendations on absolute differences among physicians, hospitals, or other levels. Seven of 12 articles that assessed reliability found that reliability was poor at the physician or hospital level due to low proportional variability and small sample sizes per unit, and cautioned that public reporting or incentives based on these measures may be inappropriate.
The proportion of variability at levels higher than patients is often found to be "low." Although low proportional variability may lead to poor measurement reliability, a number of authors further suggested that it also indicates a lack of potential for quality improvement. Few studies provided additional information to help determine whether variation was, nevertheless, clinically meaningful.

Full-text preview

Available from:
  • Source
    • "Variation in performance[43,44], quality and safety[45,46]and service usage[47]indicators among acute care providers in NSW and elsewhere has been identified. Taking into account patient and admission differences, notable inter-hospital variability in discrepancy incidence rates was evident among 80 NSW public hospitals. "
    [Show abstract] [Hide abstract] ABSTRACT: Diagnostic data routinely collected for hospital admitted patients and used for case-mix adjustment in care provider comparisons and reimbursement are prone to biases. We aim to measure discrepancies, variations and associated factors in recorded chronic morbidities for hospital admitted patients in New South Wales (NSW), Australia. Of all admissions between July 2010 and June 2014 in all NSW public and private acute hospitals, admissions with over 24 hours stay and one or more of the chronic conditions of diabetes, smoking, hepatitis, HIV, and hypertension were included. The incidence of a non-recorded chronic condition in an admission occurring after the first admission with a recorded chronic condition (index admission) was considered as a discrepancy. Poisson models were employed to (i) derive adjusted discrepancy incidence rates (IR) and rate ratios (IRR) accounting for patient, admission, comorbidity and hospital characteristics and (ii) quantify variation in rates among hospitals. The discrepancy incidence rate was highest for hypertension (51% of 262,664 admissions), followed by hepatitis (37% of 12,107), smoking (33% of 548,965), HIV (27% of 1500) and diabetes (19% of 228,687). Adjusted rates for all conditions declined over the four-year period; with the sharpest drop of over 80% for diabetes (47.7% in 2010 vs. 7.3% in 2014), and 20% to 55% for the other conditions. Discrepancies were more common in private hospitals and smaller public hospitals. Inter-hospital differences were responsible for 1% (HIV) to 9.4% (smoking) of variation in adjusted discrepancy incidences, with an increasing trend for diabetes and HIV. Chronic conditions are recorded inconsistently in hospital administrative datasets, and hospitals contribute to the discrepancies. Adjustment for patterns and stratification in risk adjustments; and furthermore longitudinal accumulation of clinical data at patient level, refinement of clinical coding systems and standardisation of comorbidity recording across hospitals would enhance accuracy of datasets and validity of case-mix adjustment.
    Full-text · Article · Jan 2016 · PLoS ONE
  • Source
    • "Our estimates from 22 clusters therefore need to be interpreted with caution, and there are further challenges when attempting to estimate variability at the clinician level as 16% to 34% of the clinicians contributed just one observation per indicator. Similar challenges in reliably estimating variability have been reported by Fung [32] and Huang [33] . We also introduced hospitals as a random term although hospitals were not from a random sample. "
    [Show abstract] [Hide abstract] ABSTRACT: Variability in processes of care and outcomes has been reported widely in high-income settings (at geographic, hospital, physician group and individual physician levels); however, such variability and the factors driving it are rarely examined in low-income settings. Using data from a cross-sectional survey undertaken in 22 hospitals (60 case records from each hospital) across Kenya that aimed at evaluating the quality of routine hospital services, we sought to explore variability in four binary inpatient paediatric process indicators. These included three prescribing tasks and use of one diagnostic. To examine for sources of variability, we examined intra-class correlation coefficients (ICC) and their changes using multi-level mixed models with random intercepts for hospital and clinician levels and adjusting for patient and clinician level covariates. Levels of performance varied substantially across indicators and hospitals. The absolute values for ICCs also varied markedly ranging from a maximum of 0.48 to a minimum of 0.09 across the models for HIV testing and prescription of zinc, respectively. More variation was attributable at the hospital level than clinician level after allowing for nesting of clinicians within hospitals for prescription of quinine loading dose for malaria (ICC = 0.30), prescription of zinc for diarrhoea patients (ICC = 0.11) and HIV testing for all children (ICC = 0.43). However, for prescription of correct dose of crystalline penicillin, more of the variability was explained by the clinician level (ICC = 0.21). Adjusting for clinician and patient level covariates only altered, marginally, the ICCs observed in models for the zinc prescription indicator. Performance varied greatly across place and indicator. The variability that could be explained suggests interventions to improve performance might be best targeted at hospital level factors for three indicators and clinician factors for one. Our data suggest that better understanding of performance and sources of variation might help tailor improvement interventions although further data across a larger set of indicators and sites would help substantiate these findings.
    Full-text · Article · Apr 2015 · Implementation Science
  • Source
    • "Furthermore, the results show that 3% of the variance in patients' mean BMI change can be attributed to differences between dietitians. This is relatively low compared to other studies in primary health care (Fung et al., 2010). Low proportional variances have often been interpreted as indicating little potential for quality improvement efforts. "
    [Show abstract] [Hide abstract] ABSTRACT: Greater insight into the effectiveness of usual dietetic care will contribute to the ongoing development of dietetic services. The present study examined the change in body mass index (BMI) in overweight patients after dietetic treatment in primary care, the sources of variability and factors associated with BMI change. This population-based observational study was based on data from a Dutch registration network of dietitians in primary health care. Data were derived from electronic medical records concerning 3960 overweight adult patients (BMI ≥ 25 kg m(-2) ) who received usual care from 32 registered dietitians between 2006 and 2012. Multilevel linear regression analyses were conducted. Patients' BMI significantly (P < 0.001) decreased by 0.94 kg m(-2) on average during treatment. An additional reduction of 0.8 kg m(-2) was observed in patients treated for longer than 6 months. BMI decreased by 0.06 kg m(-2) for each additional unit in initial BMI above 31.6. Most (97%) variability in BMI change was attributed to patients and 3% to dietitians. Part of the variance between patients (11%) and dietitians (30%) was explained by patient sociodemographic characteristics, nutrition-related health aspects, initial body weight and treatment duration. Dietetic treatment in primary care lowers BMI in overweight patients. Patients' change in BMI was rather similar between dietitians. Greater BMI reductions were observed in those with a high initial BMI and those treated for at least 6 months. Future research is necessary to study the long-term effects of weight loss after treatment by primary healthcare dietitians, especially because many patients drop out of treatment prematurely.
    Full-text · Article · Nov 2013 · Journal of Human Nutrition and Dietetics
Show more