National Release of the Nursing Home Quality Report Cards: Implications of Statistical Methodology for Risk Adjustment

Department of Medicine, University of California, Irvine, CA 92697, USA.
Health Services Research (Impact Factor: 2.78). 03/2009; 44(1):79-102. DOI: 10.1111/j.1475-6773.2008.00910.x
Source: PubMed


To determine how alternative statistical risk-adjustment methods may affect the quality measures (QMs) in nursing home (NH) report cards. DATA SOURCES/STUDY SETTINGS: Secondary data from the national Minimum Data Set files of 2004 and 2005 that include 605,433 long-term residents in 9,336 facilities.
We estimated risk-adjusted QMs of decline in activities of daily living (ADL) functioning using classical, fixed-effects, and random-effects logistic models. Risk-adjusted QMs were compared with each other, and with the published QM (unadjusted) in identifying high- and low-quality facilities by either the rankings or 95 percent confidence intervals of QMs.
Risk-adjusted QMs showed better overall agreement (or convergent validity) with each other than did the unadjusted versus each adjusted QM; the disagreement rate between unadjusted and adjusted QM can be as high as 48 percent. The risk-adjusted QM derived from the random-effects shrinkage estimator deviated nonrandomly from other risk-adjusted estimates in identifying the best 10 percent facilities using rankings.
The extensively risk-adjusted QMs of ADL decline, even when estimated by alternative statistical methods, show higher convergent validity and provide more robust NH comparisons than the unadjusted QM. Outcome rankings based on ADL decline tend to show lower convergent validity when estimated by the shrinkage estimator rather than other statistical methods.

Download full-text


Available from: Laurent Glance, Oct 05, 2015
1 Follower
30 Reads
  • Source
    • "Roy and Mor drew attention to measurement of quality and the importance of unbiased coding in data collection [27]. Li et al. showed the importance of adjusting for individual and facility-level characteristics when describing nursing home care quality [28]. Austin and Reeves, in a study assessing models of hospital care quality, concluded that the c-statistic was of little use to assess model fit although widely-accepted [29]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background This paper considers approaches to the question “Which long-term care facilities have residents with high use of acute hospitalisations?” It compares four methods of identifying long-term care facilities with high use of acute hospitalisations by demonstrating four selection methods, identifies key factors to be resolved when deciding which methods to employ, and discusses their appropriateness for different research questions. Methods OPAL was a census-type survey of aged care facilities and residents in Auckland, New Zealand, in 2008. It collected information about facility management and resident demographics, needs and care. Survey records (149 aged care facilities, 6271 residents) were linked to hospital and mortality records routinely assembled by health authorities. The main ranking endpoint was acute hospitalisations for diagnoses that were classified as potentially avoidable. Facilities were ranked using 1) simple event counts per person, 2) event rates per year of resident follow-up, 3) statistical model of rates using four predictors, and 4) change in ranks between methods 2) and 3). A generalized mixed model was used for Method 3 to handle the clustered nature of the data. Results 3048 potentially avoidable hospitalisations were observed during 22 months’ follow-up. The same “top ten” facilities were selected by Methods 1 and 2. The statistical model (Method 3), predicting rates from resident and facility characteristics, ranked facilities differently than these two simple methods. The change-in-ranks method identified a very different set of “top ten” facilities. All methods showed a continuum of use, with no clear distinction between facilities with higher use. Conclusion Choice of selection method should depend upon the purpose of selection. To monitor performance during a period of change, a recent simple rate, count per resident, or even count per bed, may suffice. To find high–use facilities regardless of resident needs, recent history of admissions is highly predictive. To target a few high-use facilities that have high rates after considering facility and resident characteristics, model residuals or a large increase in rank may be preferable.
    BMC Medical Research Methodology 07/2014; 14(1):93. DOI:10.1186/1471-2288-14-93 · 2.27 Impact Factor
  • Source
    • "intendeduseoftheQMistomeasurecontinencecarequality,theQMshould beadjustedforexogenousriskfactors.Giventhemultidimensionalnatureof NHcare,physicalfunctionandotherriskcomponentsofincontinencemay alsoreflectotheraspectsofcarepractice.Nonetheless,thisstudyfocuseson theincontinenceoutcomeandwedonotfeelthatoneQMcouldbeappropriatelyusedforprofilingmultipleaspectsofNHcare.Theissueofmulti- dimensionalityofNHqualityanditsimplicationsforriskadjustmentwere discussedbefore(Mukameletal.2008a). StudieshavecomparedtheCMSQMsandfurtherrisk-adjustedQMsin rankingfacilitiesandreportedthatcross-sectionalrankingstendedtodisagree (Arlingetal.2007;Mukameletal.2008a;Lietal.2009).Thesestudiescon- cludethatfurtherrisk-adjustedQMsmaybeanimprovementoverexisting QMs,assumingthatmoreextensiveriskadjustmentcansuccessfullyremove theinfluenceofcasemixonoutcomes.Ourstudyexplicitlytestedthisas- sumptionandconfirmedthattheoverallcase-miximpactcanbeminimized. However,whenwefocusedonfacilitieswithmoreextremeoutcomerankings, therisk-adjustedincontinenceQMstillcorrelated,althoughtoamuchlesser extent,withseveralcase-mixvariables(Table3).Thisraisesanissuerelatedto currentNHqualityimprovementefforts,whichtendtotargetfacilitieswith extremeperformance.Forexample,statesurveyorsmayexertmoreextensive oversightonfacilitieswithpooreroutcomes,orP4Pprogramsmaybede- signedtofinanciallyrewardfacilitieswiththebestoutcomes.Werecommend cautionindealingwiththese''extreme''facilitiesbecausetheirQMrate,even withextensiveriskadjustment,maystillbeaffectedbycasemix. "
    [Show abstract] [Hide abstract]
    ABSTRACT: To assess the impact of facility case mix on cross-sectional variations and short-term stability of the "Nursing Home Compare" incontinence quality measure (QM) and to determine whether multivariate risk adjustment can minimize such impacts. Retrospective analyses of the 2005 national minimum data set (MDS) that included approximately 600,000 long-term care residents in over 10,000 facilities in each quarterly sample. Mixed logistic regression was used to construct the risk-adjusted QM (nonshrinkage estimator). Facility-level ordinary least-squares models and adjusted R(2) were used to estimate the impact of case mix on cross-sectional and short-term longitudinal variations of currently published and risk-adjusted QMs. At least 50 percent of the cross-sectional variation and 25 percent of the short-term longitudinal variation of the published QM are explained by facility case mix. In contrast, the cross-sectional and short-term longitudinal variations of the risk-adjusted QM are much less susceptible to case-mix variations (adjusted R(2)<0.10), even for facilities with more extreme or more unstable outcome. Current "Nursing Home Compare" incontinence QM reflects considerable case-mix variations across facilities and over time, and therefore it may be biased. This issue can be largely addressed by multivariate risk adjustment using risk factors available in the MDS.
    Health Services Research 10/2009; 45(1):79-97. DOI:10.1111/j.1475-6773.2009.01061.x · 2.78 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: © 2002 Optical Society of America
Show more