Meaningful Variation in Performance: A Systematic Literature Review

Division of Research, Kaiser Permanente Medical Care Program, Oakland, CA, USA.
Medical care (Impact Factor: 3.23). 02/2010; 48(2):140-8. DOI: 10.1097/MLR.0b013e3181bd4dc3
Source: PubMed


Recommendations for directing quality improvement initiatives at particular levels (eg, patients, physicians, provider groups) have been made on the basis of empirical components of variance analyses of performance.
To review the literature on use of multilevel analyses of variability in quality.
Systematic literature review of English-language articles (n = 39) examining variability and reliability of performance measures in Medline using PubMed (1949-November 2008).
Variation was most commonly assessed at facility (eg, hospital, medical center) (n = 19) and physician (n = 18) levels; most articles reported variability as the proportion of total variation attributable to given levels (n = 22). Proportions of variability explained by aggregated levels were generally low (eg, <19% for physicians), and numerous authors concluded that the proportion of variability at a specific level did not justify targeting quality interventions to that level. Few articles based their recommendations on absolute differences among physicians, hospitals, or other levels. Seven of 12 articles that assessed reliability found that reliability was poor at the physician or hospital level due to low proportional variability and small sample sizes per unit, and cautioned that public reporting or incentives based on these measures may be inappropriate.
The proportion of variability at levels higher than patients is often found to be "low." Although low proportional variability may lead to poor measurement reliability, a number of authors further suggested that it also indicates a lack of potential for quality improvement. Few studies provided additional information to help determine whether variation was, nevertheless, clinically meaningful.

Full-text preview

Available from:
  • Source
    • "Variation in performance[43,44], quality and safety[45,46]and service usage[47]indicators among acute care providers in NSW and elsewhere has been identified. Taking into account patient and admission differences, notable inter-hospital variability in discrepancy incidence rates was evident among 80 NSW public hospitals. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Diagnostic data routinely collected for hospital admitted patients and used for case-mix adjustment in care provider comparisons and reimbursement are prone to biases. We aim to measure discrepancies, variations and associated factors in recorded chronic morbidities for hospital admitted patients in New South Wales (NSW), Australia. Of all admissions between July 2010 and June 2014 in all NSW public and private acute hospitals, admissions with over 24 hours stay and one or more of the chronic conditions of diabetes, smoking, hepatitis, HIV, and hypertension were included. The incidence of a non-recorded chronic condition in an admission occurring after the first admission with a recorded chronic condition (index admission) was considered as a discrepancy. Poisson models were employed to (i) derive adjusted discrepancy incidence rates (IR) and rate ratios (IRR) accounting for patient, admission, comorbidity and hospital characteristics and (ii) quantify variation in rates among hospitals. The discrepancy incidence rate was highest for hypertension (51% of 262,664 admissions), followed by hepatitis (37% of 12,107), smoking (33% of 548,965), HIV (27% of 1500) and diabetes (19% of 228,687). Adjusted rates for all conditions declined over the four-year period; with the sharpest drop of over 80% for diabetes (47.7% in 2010 vs. 7.3% in 2014), and 20% to 55% for the other conditions. Discrepancies were more common in private hospitals and smaller public hospitals. Inter-hospital differences were responsible for 1% (HIV) to 9.4% (smoking) of variation in adjusted discrepancy incidences, with an increasing trend for diabetes and HIV. Chronic conditions are recorded inconsistently in hospital administrative datasets, and hospitals contribute to the discrepancies. Adjustment for patterns and stratification in risk adjustments; and furthermore longitudinal accumulation of clinical data at patient level, refinement of clinical coding systems and standardisation of comorbidity recording across hospitals would enhance accuracy of datasets and validity of case-mix adjustment.
    Full-text · Article · Jan 2016 · PLoS ONE
  • Source
    • "Furthermore, the results show that 3% of the variance in patients' mean BMI change can be attributed to differences between dietitians. This is relatively low compared to other studies in primary health care (Fung et al., 2010). Low proportional variances have often been interpreted as indicating little potential for quality improvement efforts. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Greater insight into the effectiveness of usual dietetic care will contribute to the ongoing development of dietetic services. The present study examined the change in body mass index (BMI) in overweight patients after dietetic treatment in primary care, the sources of variability and factors associated with BMI change. This population-based observational study was based on data from a Dutch registration network of dietitians in primary health care. Data were derived from electronic medical records concerning 3960 overweight adult patients (BMI ≥ 25 kg m(-2) ) who received usual care from 32 registered dietitians between 2006 and 2012. Multilevel linear regression analyses were conducted. Patients' BMI significantly (P < 0.001) decreased by 0.94 kg m(-2) on average during treatment. An additional reduction of 0.8 kg m(-2) was observed in patients treated for longer than 6 months. BMI decreased by 0.06 kg m(-2) for each additional unit in initial BMI above 31.6. Most (97%) variability in BMI change was attributed to patients and 3% to dietitians. Part of the variance between patients (11%) and dietitians (30%) was explained by patient sociodemographic characteristics, nutrition-related health aspects, initial body weight and treatment duration. Dietetic treatment in primary care lowers BMI in overweight patients. Patients' change in BMI was rather similar between dietitians. Greater BMI reductions were observed in those with a high initial BMI and those treated for at least 6 months. Future research is necessary to study the long-term effects of weight loss after treatment by primary healthcare dietitians, especially because many patients drop out of treatment prematurely.
    Full-text · Article · Nov 2013 · Journal of Human Nutrition and Dietetics
  • Source
    • "These statistical adjustments are performed with the use of prior evidence on the relation between the performance measure and certain selected factors. In the field of education, research into student performance found (next to individual student variation) systematic variation between countries, neighborhoods, schools and classes (Fung et al., 2010; Meyer, 1997). Fried et al., (2002) showed that adjusting performance scores in nursing homes changed the rankings of the homes dramatically. "

    Full-text · Article · Sep 2013
Show more