Development and Validation of a Risk-
Adjustment Tool in Acute Asthma
Chu-Lin Tsai, Sunday Clark, Ashley F. Sullivan, and
Carlos A. Camargo, Jr.
Objective. To develop and prospectively validate a risk-adjustment tool in acute
Data Sources. Data were obtained from two large studies on acute asthma, the Mul-
ticenter Airway Research Collaboration (MARC) and the National Emergency De-
partment Safety Study (NEDSS) cohorts. Both studies involved 460 emergency
departments (EDs) and were performed during 1996–2001 and 2003–2006, respec-
tively. Both includedpatients aged18–54 years presenting to the ED with acute asthma.
Study Design. Retrospective cohort studies.
Data Collection. Clinicalinformationwasobtainedfrommedicalrecordreview.The
riskindexwasderivedintheMARCcohort and thenwas prospectively validatedinthe
Principle Findings. There were 3,515 patients in the derivation cohort and 3,986 in
the validation cohort. The risk index included nine variables (age, sex, current smoker,
ever admitted for asthma, ever intubated for asthma, duration of symptoms, respiratory
rate, peak expiratory flow, and number of beta-agonist treatments) and showed satis-
factory discrimination (area under the receiver operating characteristic curve, 0.75) and
calibration ( p5.30 for Hosmer–Lemeshow test) when applied to the validation cohort.
Conclusions. We developed and validated a novel risk-adjustment tool in acute
asthma. This tool can be used for health care provider profiling to identify outliers for
quality improvement purposes.
Key Words. Asthma, emergency department, hospital admission, profiling, risk
Risk adjustment is an important method in health services research, partic-
ularly when profiling provider performance and adjusting capitation-based
payment (Iezzoni et al. 1998; Majeed, Bindman, and Weiner 2001a,b; Blu-
menthal et al. 2005). A number of risk-adjustment tools have been developed
in cardiology (Krumholz et al. 1999; Hall et al. 2007), trauma (Reiter et al.
2004), and critical care (Zimmerman et al. 2006) for profiling hospital per-
rHealth Research and Educational Trust
formance. Risk-adjustment tools for acute respiratory disorders, such as acute
asthma, are very limited. Acute asthma is a common medical problem, ac-
counting for approximately 2 million emergency department (ED) visits and
500,000 hospitalizations each year (Moorman et al. 2007). Despite its impor-
tance, only a few risk indices or scoring systems for acute asthma are available
in theliterature (Rodrigo and Rodrigo 1997,1998; Cham etal.2002; Gorelick
et al. 2004; Kelly, Kerr, and Powell 2004). These indices, however, are de-
signed to risk-stratify asthmatic patients in clinical practice and include subtle
result, these indices are not well suited for risk adjustment in health services
tool for acute asthma using data from two large multicenter cohort studies. To
a potentially important outcome measure and profiled admission practices
across the EDs. Hospitalization is an important outcome in asthma because it
represents a large portion of the expenditures for asthma care, with an esti-
mated $4.7 billion spent each year (NHLBI 2007). The decision to admit,
however, can vary from hospital to hospital (Morris and Munasinghe 1994;
Ansari et al. 2003; Lougheed et al. 2006). With risk adjustment, differences in
for sicker patients are not unfairly penalized for their higher admission rates.
Moreover, by minimizing the differences in patient mix, practice profiling can
these unexplained variations could be further investigated.
Derivation Cohort: Multicenter Airway Research Collaboration (MARC). The
MARC is a division of the Emergency Medicine Network (EMNet, http://
of Emergency Medicine, Massachusetts General Hospital, Harvard Medical School, 326 Cam-
Sullivan, M.S., M.P.H. and Carlos A. Camargo, Jr., M.D., Dr.PH., are with the EMNet
Coordinating Center, Department of Emergency Medicine, Massachusetts General Hospital,
1702HSR: Health Services Research 44:5, Part I (October 2009)
www.emnet-usa.org). Details of the study design and data collection have
been published previously (Banerji et al. 2006). The MARC database
combines data from four observational cohort studies performed during
1996–2001. Using a standardized protocol, investigators at 76 U.S. EDs
provided 24hour/day coverage for a median of 2 weeks. Inclusion criteria
were physician diagnosis of acute asthma, age 18–54, and the ability to give
informed consent. Repeat visits by individual subjects were excluded.
Patients’ demographics, asthma history, and details of their current
exacerbation were obtained by ED interview. Data on ED management
and disposition were obtained using medical chart review. For those who did
not complete the ED interview (missed by investigators, refused, or other
reasons), their medical records were reviewed to capture full data on
demographics, ED presentation, ED course, as well as limited information on
asthma history. Because each of the interviewed subjects also had data
collected from their medical records, the MARC database represents all
eligible patients presenting to the ED during the study periods. For the
current analysis, we focused on the variables taken from medical records.
Validation Cohort: Asthma Component of the National Emergency Department Safety
Study (NEDSS). The NEDSS is a large, multicenter study designed to
characterize organizational- and clinician-related factors associated with the
occurrence of errors in EDs. Details of the study design and data collection
have been published previously (Sullivan et al. 2007). In brief, NEDSS was
also coordinated by EMNet and recruited EDs by directly inviting sites
affiliated with EMNet; EDs not yet affiliated with EMNet were invited
through postings on emergency medicine listservs and presentations at
national emergency medicine meetings. Three clinical conditions were
selected and examined in the NEDSS: acute myocardial infarction,
dislocations, and acute asthma. The current analysis examined the asthma
component (Tsai et al. 2009). Using a standardized data abstraction tool,
trained research personnel at 63 U.S. EDs abstracted data from randomly
selected ED visits for acute asthma during 2003–2006. The visits were
identifiedby using International Classification of Diseases, Ninth Revision, Clinical
Modification (ICD-9-CM) codes 493.xx. Inclusion criteria were age 14–54
years and a history of asthma before the index visit. The following visits were
excluded: repeat visits; transfer visits; patient visits with a history of chronic
obstructive pulmonary disease, emphysema, or chronic bronchitis; or visits
not prompted, in large part, by asthma exacerbation. Similar to MARC, data
Risk-Adjustment Tool in Acute Asthma1703
abstraction focused on baseline patient characteristics, past asthma history,
ED presentation, management, and disposition. One hospital was prohibited
thus the risk index cannot be calculated for this site. Thus, we dropped this
site from the NEDSS cohort, leaving 62 EDs in the NEDSS analysis.
Information on Spirometry
Peak expiratory flow (PEF) was recorded in l/minute and expressed as the
absolute value; no predicted values are presented due to lack of the patient’s
height. Severity of acute asthma was classified according to the initial PEF as
follows: mild, 3001 for women, 4001 for men; moderate, 200–299 for
women, 250–399 formen; severe,120–199 forwomen, 150–249 formen; and
very severe, o120 for women, o150 for men. The absolute PEF cutoffs
represented approximately 70, 40, and 25 percent predicted, respectively, for
a typical adult woman and man (Radeos and Camargo 2004).
The outcome measure was hospital admission, which was defined as admis-
sion to an inpatient unit, observation unit, or intensive care unit. We chose
hospital admission as the most relevant severity measure because mortality is
very rare in acute asthma.
All analyses were performed using Stata 10.0 (StataCorp, College Station,
TX). Summary statistics are presented as proportions (with 95 percent con-
fidence intervals [CI]), means (with standard deviations), or medians (with
interquartile ranges). All p values are two-sided, with po.05 considered sta-
Derivation and Validation of the Risk-Adjustment Tool. Multivariable logistic
from the MARC database. Model variables had to be readily available in the
medical record and were selected a priori based on the review of the medical
literature (Rodrigo and Rodrigo 1997, 1998; Emerman et al. 1999; Kelly,
Powell, and Kerr 2002; Weber et al. 2002; Kelly, Kerr, and Powell 2004) and
clinical experience. The variable domains included the following: de-
mographics, chronic asthma-related factors, ED presentation and severity,
1704HSR: Health Services Research 44:5, Part I (October 2009)
and ED course. To determine the functional form used for continuous
predictors, we grouped the predictor into bins of equal width and checked if
log odds of admission increased or decreased linearly. If the linearity
assumption did not hold, dummy coded categorical variables were generated
to characterize the dose–response relationship. Variables with missing data
were dummy coded using the missing indicator method (Miettinen 1985).
This method of modeling missing data assumes data are missing at random.
The performance of the model was evaluated by discrimination and
calibration. The discriminatory power was quantified by determining the
area under the receiver operating characteristic (ROC) curve. The calibration
was measured by comparing predicted versus observed admissions in each
decile of admission probability using the Hosmer–Lemeshow goodness-of-fit
test (Hosmer and Lemeshow 2000). All odds ratios (ORs) are presented with
95 percent CI. After development of the risk index in the MARC cohort, the
regression coefficients were retained and prospectively validated in the
NEDSS cohort. The performance of the index, including discrimination and
calibration, was re-evaluated.
Sensitivity Analysis. Because PEF measurements were not available for all
patients and that the modeling for missing data requires assumptions, we
reduced the numberof covariates in the model by omitting PEF and repeated
the analyses using this reduced model.
Example of Profiling Admission Practices and Identifying Outliers. The risk-
adjustment tool can be used for profiling many severity-related outcome
paper, we demonstrated profiling admission practices. There are at least two
the ready-made approach from the simple logistic regression model (Ivanov,
Tu, and Naylor 1999). This method uses the validated beta coefficients as
weights and applies them to individual patient data to obtain expected
probabilities (p) of admission:
padmission¼ expðb0þ b1X1þ b2X2þ ??? þ bnXnÞ=½1 þ expðb0þ b1X1
þ b2X2þ ??? þ bnXnÞ?
These individual probabilities are then averaged at the hospital level to give
for each hospital are calculated by dividing the hospital’s actual admission
Risk-Adjustment Tool in Acute Asthma1705
rates by its expected value and then multiplying that by the hospital-wide
A more sophisticated approach is to use hierarchical modeling, which
takes into account the potential for clustered observations within hospitals
(Krumholz et al. 2006a; Tsai 2009), as demonstrated in this paper. We
performed a random intercept, two-level hierarchical logistic regression
model. This model included the fixed effects of the patient-level covariates
comprising the risk index, plus a hospital-level random intercept (Rabe-
but re-estimated the beta coefficients according to the study population and
hierarchical model specification; for example, the beta coefficients in model
2 are different from those in model 1:
LogoddsðpijÞ ¼ b0jþ b1X1ijþ b2X2ijþ ??? þ bnXnij
b0j¼ b0þ tjwheretj? Nð0; s2Þ
where pijdenotes the probability of admission for the ith patient treated at the
jth hospital; X1ijthrough Xnijdenote patient characteristics. The above model
incorporates a normally distributed hospital random effect (tj). This hospital-
specific random effect is the logarithm of the OR of admission at the given
hospital compared with a hospital with an averageadmission rate in the study
population, after adjusting for patient mix. The patients treated at hospitals
with positive random effects have greater odds of admission than patients
treated at a hospital with an average admission rate.
For each hospital, an estimate of that hospital’s random effects was
computed, as was its standard error. To identify the outliers in admission
practice, hospitals were ranked by their point estimates of random effects.
Ninety-five percent CIs were plotted around the point estimates (aka
caterpillar plot). Those hospitals whose 95 percent CI lay entirely above
zero were classified as having significantly higher-than-average admission
rates, while those hospitals whose 95 percent CI lay entirely below zero were
classified ashaving significantlylower-than-averageadmission rates (DeLong
et al. 1997). The impact of risk adjustment on hospital rankings was assessed
by changes in tertile ranking before and after risk adjustment, as well as a
weighted k coefficient of agreement.
If one is interested in hospital characteristics associated with admission
rather than identifying the hospital outliers, one could enter hospital
characteristics as fixed-effect parameters into model 2 (i.e., H1j–Hkjas in
1706HSR: Health Services Research 44:5, Part I (October 2009)
LogoddsðPijÞ ¼ b0jþ b1X1ijþ b2X2ijþ ??? þ bnXnij
b0j¼ b0þ a1H1jþ a2H2jþ ??? þ akHkjþ tjwheretj? Nð0; s2Þ
A key informant survey was distributed at each site in NEDSS to collect data
refit the hierarchical model as follows: number of beds in the ED, annual visit
volume, geographic regions (Northeast, South, Midwest, and West),
affiliation with an emergency medicine residency program, number of ED
bed conference,’’ and whether ED attending physicians had admitting
There were 3,515 patients with acute asthma in the MARC derivation cohort
and 3,986 in the NEDSS validation cohort. The patients in the derivation and
validation cohorts were quite similar (Table 1). The median age was between
30 and 40 years for both cohorts, and there were more women than men in
intubation for asthma, compared with the NEDSS cohort. Acute ED presen-
as having moderate-to-severe asthma according to the initial PEF. Admission
rates were 21 and 19 percent in the derivation and validation cohorts, re-
Derivation of the Risk Index
The model from the derivation cohort was comprised of nine variables, in-
cluding demographics, chronic asthma-related factors, acuity at ED presen-
admission for asthma, higher respiratory rate and lower PEF at ED presen-
tation, and more intensive b-agonist treatments were independently associ-
ated with an increased risk of hospital admission. In contrast, shorter duration
of symptoms was associated with a decreased risk of admission. The area
under the ROC curve for the model from the derivation cohort was 0.75; the
model fit was satisfactory (p5.39 for Hosmer–Lemeshow test) (Table 3).
Risk-Adjustment Tool in Acute Asthma1707
Validation of the Risk Index
satisfactory discriminatory ability (area under the ROC, 0.75) and calibration
(p5.30 for Hosmer–Lemeshow test) (Table 3). The satisfactory calibration
was evident in the plot of observed versus predicted probabilities of admis-
sion. In all deciles of admission probability, the predicted probabilities of
admission were fairly consistent with actual risks of admission (Figure 1).
When omitting PEF from the MARC model, the discriminatory ability of the
was maintained (p5.41 for Hosmer–Lemeshow test) (Table 3). The reduced
Characteristics of the Patients in the Derivation and Validation
MARC Derivation Cohort
NEDSS Validation Cohort
Number of patients
Age (years), median (IQR)
Current smoker (%)
Chronic asthma-related factors
Ever admitted for asthma (%)
Ever intubated for asthma (%)
Duration of symptoms o24 hours (%)
Initial respiratory rate (breaths/
minute), median (IQR)
Initial PEF (L/minute), median (IQR)n
Severity based on initial PEF (%)
Number of inhaled b-agonist treatments during 1st hour
22 (20–26)20 (18–24)
206 (150–280) 240 (170–300)
nAvailable for 2,560 patients in MARC, and for 1,837 patients in NEDSS.
wEach nebulizer treatment was counted as equivalent to six puffs from a metered-dose inhaler.
ED, emergency department; IQR, interquartile range; MARC, Multicenter Airway Research Col-
laboration; NEDSS, National Emergency Department Safety Study; PEF, peak expiratory flow.
1708 HSR: Health Services Research 44:5, Part I (October 2009)
eight-variable model still performed satisfactorily when applied to the vali-
dation cohort (Table 3).
Example of Profiling Admission Practices and Identifying Outliers
Results foridentifying hospitals asoutliers in admissionpracticesareshown in
Figure S1. After adjusting for patient mix, nine hospitals were identified as
having significantly lower admission rates, while 18 hospitals were identified
as having significantly higher admission rates in the NEDSS sample.
Hospitals were ranked according to the random effects obtained from
the hierarchical model. After adjusting for patient mix, there were significant
changes in the tertile rankings, with all the changes occurring between ad-
jacent categories (Table S1). The weighted k coefficient showed only a mod-
erate agreement of hospital rankings before and after risk adjustment
(unweighted k coefficient was 0.47 and weighted k with linear weighting
sion in the Derivation Cohort
Multivariable Model of Factors Associated with Hospital Admis-
VariableOdds Ratio (95% Confidence Interval) b Coefficients
Ever admitted for asthma
Ever intubated for asthma (%)
Duration of symptoms o24 hours
Initial respiratory rate
Severity based on initial PEF (%)
Number of b-agonist treatments during 1st hour
nOdds ratio per increase in 5breaths/minute.
PEF, peak expiratory flow.
Risk-Adjustment Tool in Acute Asthma 1709
Further inclusion of ED characteristics in the hierarchical model re-
vealed that only the number of ED beds was independently, positively as-
sociated with hospitalization (OR per 1-bed increase, 1.05; 95 percent CI,
1.01–1.09), after adjusting for patient mix.
Using data from two large cohorts, we developed and prospectively validated
a risk-adjustment tool for acute asthma. We also demonstrated that this tool
can be used for profiling admission practices across hospitals. Given its va-
lidity, we believe that this tool may have broader uses, particularly in mon-
itoring and reporting performance of hospitals and health care providers, as
well as in reimbursement control.
A number of studies have proposed hospital admission as a proxy for
severity of illness in acute care settings and have developed risk adjustment
models using admission as an outcome measure (Chamberlain et al. 2004;
Gorelick et al. 2007). These models, however, are ‘‘generic’’ in nature and
few asthma-specific risk indices or scoring systems are available (Rodrigo and
Rodrigo 1997, 1998; Cham et al. 2002; Gorelick et al. 2004; Kelly, Kerr, and
Powell 2004). However, as mentioned before, these tools either utilize re-
peated measurements of lung function (Rodrigo and Rodrigo 1998; Kelly,
Kerr, and Powell 2004) or incorporate subtle physical findings (e.g., accessory
Table3: Comparison of Discrimination and Calibration of the Models
MARC ModelMARC Model Omitting PEF
nA value of p ? .05 indicates satisfactory calibration.
MARC, Multicenter Airway Research Collaboration; NEDSS, National Emergency Department
Safety Study; PEF, peak expiratory flow; ROC, receiver operating characteristic.
1710HSR: Health Services Research 44:5, Part I (October 2009)
muscle use) (Rodrigo and Rodrigo 1997; Cham et al. 2002; Gorelick et al.
2004), both of which are infrequently documented in the medical record.
Risk-adjustment models should be developed and validated in different
samples to assess robustness because external validation is the true test of a
predictive model (Harrell, Lee, and Mark 1996; Krumholz et al. 2006a). Al-
though the NEDSS patients seemed to be less ill compared with the MARC
patients, the risk index retained satisfactory discrimination and calibration
when applied to the NEDSS data. The stability of the model over time sup-
ports the validity of the nine variables in the index. It is possible that a simpler
risk-adjustment tool based on administrative data will be developed in the
future, and this medical record–based model may be used to validate the
administrative claimsmodel,ashealthservicesresearchers have doneinheart
failure and acute myocardial infarction (Krumholz et al. 2006b,2006c).
We have shown that the risk index can be incorporated into the hier-
archical model for benchmarking admission practices across hospitals. By
inspecting the ‘‘caterpillar plot,’’ significant deviations from the average
should prompt review of the medical practices (utilization management), es-
of predicted admission probability. The difference between the circles and the
dotted line represents over- or underprediction of actual risk of admission.
MARC, Multicenter Airway Research Collaboration; NEDSS, National Emergency
Department Safety Study.
Risk-Adjustment Tool in Acute Asthma 1711
pecially in the hospitals with the highest deviations from the reference. For
those hospitals that potentially overadmit patients, payment for unnecessary
services may be denied to avoid a waste of inpatient resources. For those
hospitals that potentially fail to admit patients when necessary, physicians’ re-
education and feedback on their practice patterns may beneededto minimize
adverse events among patients discharged from the ED.
Because the results of performance ranking (i.e., report card) have pro-
found effects on hospitals and health care providers (Shahian et al. 2005), it is
critically important that the risk-adjustment tool is updated, transparent, and
accountable, and that the statistical methodology for profiling is appropriate
(Tsai 2009). Some studies have shown that using hierarchical models may
avoid false outlier classification and may result in more accurate estimates of
provider performance (Shahian et al. 2001, 2005). With the use of our val-
idated risk index and the hierarchical model, provider profiling for acute
asthma would be more credible.
This study has some potential limitations. First, unlike risk-adjustment
tools derived from administrative data, the risk index requires medical record
abstraction. Although it includes more clinical information, it can be costly.
However, with the advances in information technology, electronic medical
records may provide a more efficient way to capture the information needed
for this index. Second, we used hospitalization as an outcome measure to
demonstrate the utility of the risk-adjustment tool. The decision to admit,
however, is influenced by many other factors in addition to disease severity,
such as patient preference and the availability of hospital beds (Wennberg
2002). These unwarranted variations in practice would require a closer in-
spection of medical records to determine the appropriateness of admission
decisions. In this context, the risk-adjustment tool helps identify outliers to
mitigate the burdenassociatedwithfull utilization review. Moreover, thisrisk-
adjustment tool canbeusedtolookatotherseverity-relatedoutcomes,suchas
costs and length of stay. Third, this risk index was not designed for risk strat-
ification in clinical practice. Rather, it is intended to be applied to groups of
patients at the hospital or provider level for the purposes of risk adjustment.
Finally, the EDs that composed our samples are predominantly urban, ac-
ademically affiliated hospitals. The applicability of this index to other insti-
tutions will require additional studies.
In summary, we developed and prospectively validated a novel risk-
adjustment tool in acute asthma. The tool can be used for profiling practices
improvement or reimbursement control. For policymakers, validated risk-
1712 HSR: Health Services Research 44:5, Part I (October 2009)
adjustment tools and appropriate statistical methodology increase the likeli-
hood of correct inferences and sound policies. For health care providers,
receiving regular feedback on practices should help improve decision making
and achieve a more cost-effective practice.
Joint Acknowledgment/Disclosure Statement: This investigator-initiated study was
supported by a research grant from Critical Therapeutics (Lexington, MA),
themanuscript,ordecision topublish.The underlyingstudiesweresupported
by unrestricted grants from GlaxoSmithKline (Research Triangle Park, NC)
and R01 HS-13099 from the Agency for Healthcare Research and Quality
MD). The authors thank the participating investigators for their ongoing ded-
ication to emergency medicine and patient safety research (full list in the
online supplemental material).
Disclosures: Dr. Camargohas received financial support from a variety of
groups for participation in conferences, consulting, and medical research;
recentindustry sponsorswithaninterest in asthmawere AstraZeneca, Critical
Therapeutics, Dey, Genentech, GSK, Merck, Novartis, Respironics, and
Schering-Plough. Other authors have no conflicts of interest to disclose.
Ansari, Z., M. M. Haby, T. Henderson, F. Cicuttini, and M. J. Ackland. 2003. ‘‘Trends
and Geographic Variations in Hospital Admissions for Asthma in Victoria. Op-
portunities forTargeted Interventions.’’Australian Family Physician32(4): 286–8.
Banerji, A., S. Clark, M. Afilalo, M. P. Blanda, R. K. Cydulka, and C. A. Camargo, Jr.
2006. ‘‘Prospective Multicenter Study of Acute Asthma in Younger Versus
Older Adults Presenting to the Emergency Department.’’ Journal of American
Geriatrics Society 54 (1): 48–55.
Blumenthal, D., J. S. Weissman, M. Wachterman, E. Weil, R. S. Stafford, J. M. Perrin,
T. G. Ferris, K. Kuhlthau, R. Kaushal, and L. I. Iezzoni. 2005. ‘‘The Who, What,
and Why of Risk Adjustment: A Technology on the Cusp of Adoption.’’ Journal
of Health Politics, Policy and Law 30 (3): 453–73.
Risk-Adjustment Tool in Acute Asthma1713
Respiratory Acidosis during Exacerbation of Asthma and Chronic Obstructive
Pulmonary Disease.’’ European Journal of Emergency Medicine 9 (3): 225–32.
Chamberlain, J. M., K. M. Patel, M. M. Pollack, A. Brayer, C. G. Macias, P. Okada,
J. E. Schunk, and Collaborative Research Committee of the Emergency
Medicine Section of the American Academy of Pediatrics. 2004. ‘‘Recalibration
of the Pediatric Risk of Admission Score Using a Multi-Institutional Sample.’’
Annals of Emergency Medicine 43 (4): 461–8.
DeLong, E. R., E. D. Peterson, D. M. DeLong, L. H. Muhlbaier, S. Hackett, and D. B.
Mark. 1997. ‘‘Comparing Risk-Adjustment Methods for Provider Profiling.’’
Statistics in Medicine 16 (23): 2645–64.
Emerman, C. L., P. G. Woodruff, R. K. Cydulka, M. A. Gibbs, C. V. Pollack, Jr., and
C. A. Camargo, Jr. 1999. ‘‘Prospective Multicenter Study of Relapse Following
Treatment for Acute Asthma Among Adults Presenting to the Emergency De-
partment. MARC Investigators. Multicenter Asthma Research Collaboration.’’
Chest 115 (4): 919–27.
Emergency Assessment Tool (RePEAT): A Severity Index for Pediatric Emer-
gency Care.’’ Academic Emergency Medicine 14 (4): 316–23.
of a Novel Clinical Score, the Pediatric Asthma Severity Score (PASS), in the
Evaluation of Acute Asthma.’’ Academic Emergency Medicine 11 (1): 10–8.
Hall, B. L., M. Hirbe, B. Waterman, S. Boslaugh, and W. C. Danagan. 2007. ‘‘Com-
parison of Mortality Risk Adjustment Using a Clinical Data Algorithm (Amer-
ican College of Surgeons National Surgical Quality Improvement Program) and
an Administrative Data algorithm (Solucient) at the Case Level Within a Single
Institution.’’ Journal of the American College of Surgeons 205 (6): 767–77.
Harrell, F. E. Jr., K. L. Lee, and D. B. Mark. 1996. ‘‘Multivariable Prognostic Models:
Issues in Developing Models, Evaluating Assumptions and Adequacy, and
Measuring and Reducing Errors.’’ Statistics in Medicine 15 (4): 361–87.
Hosmer, D. W., and S. Lemeshow. 2000. Applied Logistic Regression. 2d Edition. New
Iezzoni, L.I., J.Z.Ayanian,D.W. Bates,andH. R.Burstin.1998. ‘‘PayingMore Fairly
for Medicare Capitated Care.’’ New England Journal of Medicine 339 (26): 1933–8.
Ivanov, J., J. V. Tu, and C. D. Naylor. 1999. ‘‘Ready-Made, Recalibrated, or Remod-
eled? Issues in the Use of Risk Indexes for Assessing Mortality after Coronary
Artery Bypass Graft Surgery.’’ Circulation 99 (16): 2098–104.
Kelly, A. M., D. Kerr, and C. Powell. 2004. ‘‘Is Severity Assessment after One Hour of
Treatment Better for Predicting the Need for Admission in Acute Asthma?’’
Respiratory Medicine 98 (8): 777–81.
Kelly, A. M., C. Powell, and D. Kerr. 2002. ‘‘Patients with a Longer Duration of
Symptoms of Acute Asthma Are More Likely to Require Admission to Hos-
pital.’’ Emergency Medicine (Fremantle) 14 (2): 142–5.
Krumholz, H. M., R. G. Brindis, J. E. Brush, D. J. Cohen, A. J. Epstein, K. Furie, G.
Howard, E. D. Peterson, S. S. Rathore, S. C. Smith, Jr., J. A. Spertus, Y. Wang,
1714HSR: Health Services Research 44:5, Part I (October 2009)
and S. L. Normand. 2006a. ‘‘Standards for Statistical Models Used for Public
Reporting of Health Outcomes: An American Heart Association Scientific
Statement from the Quality of Care and Outcomes Research Interdisciplinary
Writing Group: Cosponsored by the Council on Epidemiology and Prevention
and the Stroke Council. Endorsed by the American College of Cardiology
Foundation.’’ Circulation 113 (3): 456–62.
Krumholz, H. M., J. Chen, Y. Wang, M. J. Radford, Y. T. Chen, and T. A. Marciniak.
1999. ‘‘Comparing AMI Mortality among Hospitals in Patients 65 Years of Age
and Older: Evaluating Methods of Risk Adjustment.’’ Circulation 99 (23): 2986–
Krumholz,H.M.,Y.Wang, J.A.Mattera,Y.Wang,L.F.Han,M. J.Ingber,S.Roman,
and S. L. Normand. 2006b. ‘‘An Administrative Claims Model Suitable for
Profiling Hospital Performance Based on 30-Day Mortality Rates among Pa-
tients with an acute Myocardial Infarction.’’ Circulation 113 (13): 1683–92.
—— —— ——. 2006c. ‘‘An Administrative Claims Model Suitable for Profiling Hospital Per-
formance Based on 30-Day Mortality Rates among Patients with Heart Failure.’’
Circulation 113 (13): 1693–701.
Lougheed, M. D., N. Garvey, K. R. Chapman, L. Cicutto, R. Dales, A. G. Day, W. M.
Hopman, M. Lam, M. R. Sears, K. Szpiro, T. To, N. A. Patterson; Ontario
Respiratory Outcomes Research Network. 2006. ‘‘The Ontario Asthma Re-
gional Variation Study: Emergency Department Visit Rates and the Relation to
Hospitalization Rates.’’ Chest 129 (4): 909–17.
Majeed, A., A. B. Bindman, and J. P. Weiner. 2001a. ‘‘Use of Risk Adjustment in
Setting BudgetsandMeasuringPerformanceinPrimary Care I: HowIt Works.’’
British Medical Journal 323 (7313): 604–7.
—— —— ——. 2001b. ‘‘Use of Risk Adjustment in Setting Budgets and Measuring Perfor-
mance in Primary Care II: Advantages, Disadvantages, and Practicalities.’’ Brit-
ish Medical Journal 323 (7313): 607–10.
Miettinen, O. S. 1985. Theoretical Epidemiology: Principles of Occurrence Research in Med-
icine. New York: Wiley.
Moorman, J.E.,R.A.Rudd,C.A. Johnson,M.King,P.Minor,C.Bailey,M.R.Scalia,
L. J. Akinbami, and Centers for Disease Control and Prevention (CDC). 2007.
‘‘National Surveillance for Asthma–United States, 1980–2004.’’ Morbidity and
Mortality Weekly Report Surveillance Summary 56 (8): 1–54.
Morris, R. D., and R. L. Munasinghe. 1994. ‘‘Geographic Variability in Hospital Ad-
mission Rates for Respiratory Disease among the Elderly in the United States.’’
Chest 106 (4): 1172–81.
NHLBI. 2007. ‘‘Morbidity & Mortality: 2007 Chart Book on Cardiovascular, Lung,
and Blood Diseases’’ [accessed June 23, 2009]. Available at http://nhlbi.nih.
Rabe-Hesketh, S., and A. Skrondal. 2005. Multilevel and Longitudinal Modeling Using
Stata. College Station, TX.: Stata Press.
Radeos, M. S., and C. A. Camargo, Jr. 2004. ‘‘Predicted Peak Expiratory Flow:
Differences across Formulae in the Literature.’’ American Journal of Emergency
Medicine 22 (7): 516–21.
Risk-Adjustment Tool in Acute Asthma 1715
Reiter, A., W. Mauritz, B. Jordan, T. Lang, A. Polzl, L. Pelinka, and P. G. Metnitz.
SAPS Score.’’ Journal of Trauma 57 (2): 375–80.
Rodrigo, G., and C. Rodrigo. 1998. ‘‘Early Prediction of Poor Response in
Acute Asthma Patients in the Emergency Department.’’ Chest 114 (4): 1016–
—— —— ——. 1997. ‘‘A New Index for Early Prediction of Hospitalization in Patients with
Acute Asthma.’’ American Journal of Emergency Medicine 15 (1): 8–13.
Shahian, D. M., S. L. Normand, D. F. Torchiana, S. M. Lewis, J. O. Pastore, R. E.
Kuntz, and P. I. Dreyer. 2001. ‘‘Cardiac Surgery Report Cards: Comprehensive
Review and Statistical Critique.’’ Annals of Thoracic Surgery 72 (6): 2155–68.
Shahian, D. M., D. F. Torchiana, R. J. Shemin, J. D. Rawn, and S. L. Normand. 2005.
‘‘Massachusetts Cardiac Surgery Report Card: Implications of Statistical Meth-
odology.’’ Annals of Thoracic Surgery 80 (6): 2106–13.
Sullivan, A. F., C. A. Camargo, Jr., P. D. Cleary, J. A. Gordon, E. Guadagnoli, R.
Kaushal, D. J. Magid, S. R. Rao, and D. Blumenthal 2007. ‘‘The National
Emergency Department Safety Study: Study Rationale and Design.’’ Academy of
Emergency Medicine 14 (12): 1182–9.
Tsai, C. L. 2009. ‘‘Analyzing Patient Case Mix and Hospital Rankings.’’ Journal of
American Medical Association 301 (11): 1125; author reply.
Tsai, C. L., A. F. Sullivan, J. A. Gordon, R. Kaushal, D. J. Magid, D. Blumenthal, and
C. A. Camargo, Jr. 2009. ‘‘Quality of Care for Acute Asthma in 63 US Emer-
gency Departments.’’ Journal of Allergy and Clinical Immunology 123 (2): 354–61.
and C. A. Camargo, Jr. 2002. ‘‘A Prospective Multicenter Study of Factors As-
sociated with Hospital Admission among Adults with Acute Asthma.’’ American
Journal of Medicine 113 (5): 371–8.
Wennberg, J. E. 2002. ‘‘Unwarranted Variations in Healthcare Delivery: Implications
for Academic Medical Centres.’’ British Medical Journal 325 (7370): 961–4.
Zimmerman, J. E., A. A. Kramer, D. S. McNair, and F. M. Malila. 2006. ‘‘Acute
Physiology and Chronic Health Evaluation (APACHE) IV: Hospital Mortality
Assessment for Today’s Critically Ill Patients.’’ Critical Care Medicine 34 (5):
Additional supporting information may be found in the online version of this
Appendix SA1: Author Matrix.
Appendix SA2. Full Acknowledgments.
1716HSR: Health Services Research 44:5, Part I (October 2009)
Figure S1. Profiling Hospital Admissions. ED-level random intercepts Download full-text
(log of the OR for admission) and their 95 percent confidence intervals, ob-
tained by the hierarchical model, are used to identify the outliers of hospital
admission practices. The black circles represent the hospital outliers that have
significantly lower or higher admission practices than an average hospital,
significantly different from an average hospital in the study population.
Rankings of Admission Practices in the Validation Cohort.
Please note: Wiley-Blackwell is not responsible for the content or func-
tionality of any supporting materials supplied by the authors. Any queries
(other than missing material) should be directed to the corresponding author
for the article.
Risk-Adjustment Tool in Acute Asthma 1717