Performance Profiling in Primary Care: Does the Choice of Statistical Model Matter?

Institute of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, the Netherlands.
Medical Decision Making (Impact Factor: 3.24). 08/2013; 34(2). DOI: 10.1177/0272989X13498825
Source: PubMed

ABSTRACT Background. Profiling is increasingly being used to generate input for improvement efforts in health care. For these efforts to be successful, profiles must reflect true provider performance, requiring an appropriate statistical model. Sophisticated models are available to account for the specific features of performance data, but they may be difficult to use and explain to providers. Objective. To assess the influence of the statistical model on the performance profiles of primary care providers. Data Source. Administrative data (2006-2008) on 2.8 million members of a Dutch health insurer who were registered with 1 of 4396 general practitioners. Methods. Profiles are constructed for 6 quality measures and 5 resource use measures, controlling for differences in case mix. Models include ordinary least squares, generalized linear models, and multilevel models. Separately for each model, providers are ranked on z scores and classified as outlier if belonging to the 10% with the worst or best performance. The impact of the model is evaluated using the weighted kappa for rankings overall, percentage agreement on outlier designation, and changes in rankings over time. Results. Agreement among models was relatively high overall (kappa typically >0.85). Agreement on outlier designation was more variable and often below 80%, especially for high outliers. Rankings were more similar for processes than for outcomes and expenses. Agreement among annual rankings per model was low for all models. Conclusions. Differences among models were relatively small, but the choice of statistical model did affect the rankings. In addition, most measures appear to be driven largely by chance, regardless of the model that is used. Profilers should pay careful attention to the choice of both the statistical model and the performance measures.

3 Reads
  • Source
    • "Ieva and Paganoni successfully used mixed effect models and funnel plots in hospital readmission rates [30]. Most recently, Eijkenaar and van Vliet compared rankings of quality, not of LTC facilities but of primary care providers, using many different statistical models to identify outliers [31]. They observed similar results from the models but very varied outliers and rankings, notably that the models better detected high-performing providers than low-performing providers. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background This paper considers approaches to the question “Which long-term care facilities have residents with high use of acute hospitalisations?” It compares four methods of identifying long-term care facilities with high use of acute hospitalisations by demonstrating four selection methods, identifies key factors to be resolved when deciding which methods to employ, and discusses their appropriateness for different research questions. Methods OPAL was a census-type survey of aged care facilities and residents in Auckland, New Zealand, in 2008. It collected information about facility management and resident demographics, needs and care. Survey records (149 aged care facilities, 6271 residents) were linked to hospital and mortality records routinely assembled by health authorities. The main ranking endpoint was acute hospitalisations for diagnoses that were classified as potentially avoidable. Facilities were ranked using 1) simple event counts per person, 2) event rates per year of resident follow-up, 3) statistical model of rates using four predictors, and 4) change in ranks between methods 2) and 3). A generalized mixed model was used for Method 3 to handle the clustered nature of the data. Results 3048 potentially avoidable hospitalisations were observed during 22 months’ follow-up. The same “top ten” facilities were selected by Methods 1 and 2. The statistical model (Method 3), predicting rates from resident and facility characteristics, ranked facilities differently than these two simple methods. The change-in-ranks method identified a very different set of “top ten” facilities. All methods showed a continuum of use, with no clear distinction between facilities with higher use. Conclusion Choice of selection method should depend upon the purpose of selection. To monitor performance during a period of change, a recent simple rate, count per resident, or even count per bed, may suffice. To find high–use facilities regardless of resident needs, recent history of admissions is highly predictive. To target a few high-use facilities that have high rates after considering facility and resident characteristics, model residuals or a large increase in rank may be preferable.
    BMC Medical Research Methodology 07/2014; 14(1):93. DOI:10.1186/1471-2288-14-93 · 2.27 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Healthcare providers are often evaluated by studying variability in their indicators. However, the usefulness of this analysis may be limited if we do not distinguish the variability attributable to health professionals and organizations from that associated with their patients.Our objectives are to describe the main process and outcome indicators of primary healthcare services, analyzing the contribution to variability in these indicators from different levels: individual, health professional, health center, and health district.This is a cross-sectional study that includes all.All the individuals covered by the public Basque Health Service (children [age 0-13], n = 247,493; adults [≥14 years old], n = 1,959,682) over a 12-month period.We calculated the number of visits to primary care doctors, number of referrals, prescription costs, and potentially avoidable hospitalizations for ambulatory care sensitive conditions (ACSCs). Using multilevel analysis, we determined the percentage of variance attributable to each level.After adjusting for the characteristics of patients (demographic, socioeconomic, and morbidity), doctors (panel size), health center (size, staff satisfaction, demographic structure of the community), and health district, the variance in the indicators was mainly attributable to differences between patients, independently of the attending health professional, the center, or the healthcare organization, both in children (94.21% for visits to the doctor; 96.66% for referrals; 98.57% for prescription costs; 90.02% for potentially avoidable hospitalizations for ACSCs) and in adults (88.10%; 96.26%; 97.92%; and 93.77%, respectively).The limited contribution of health professionals and organizations to variability in indicators should be taken into account when performing evaluations and planning quality improvement strategies.
    Medicine 08/2015; 94(31):e1314. DOI:10.1097/MD.0000000000001314 · 5.72 Impact Factor