Psychometric Properties of the Consumer Assessment of Healthcare Providers and Systems (CAHPS (R)) Clinician and Group Adult Visit Survey

Article (PDF Available)inMedical care 50 Suppl(11):S28-34 · November 2012with79 Reads
DOI: 10.1097/MLR.0b013e31826cbc0d · Source: PubMed
Abstract
: The Consumer Assessment of Healthcare Providers and Systems (CAHPS) Clinician and Group Adult Visit Survey enables patients to report their experiences with outpatient medical offices. : To evaluate the factor structure and reliability of the CAHPS Clinician and Group (CG-CAHPS) Adult Visit Survey. : Data from 21,318 patients receiving care in 450 clinical practice sites collected from March 2010 to December 2010 were analyzed from the CG-CAHPS Database. RESEARCH DESIGN AND PARTICIPANTS:: Individual level and multilevel confirmatory factor analyses were used to examine CAHPS survey responses at the patient and practice site levels. We also estimated internal consistency reliability and practice site level reliability. Correlations among multi-item composites and correlations between the composites and 2 global rating items were examined. : Scores on CG-CAHPS composites assessing Access to Care, Doctor Communication, Courteous/Helpful Staff, and 2 global ratings of whether one would Recommend their Doctor, and an Overall Doctor Rating. : Analyses provide support for the hypothesized 3-factor model assessing Access to Care, Doctor Communication, and Courteous/Helpful Staff. In addition, the internal consistency reliabilities were ≥0.77 and practice site level reliabilities for sites with >4 clinicians were ≥0.75. All composites were positively and significantly correlated with the 2 global rating items, with Doctor Communication having the strongest relationship with the global ratings. : The CG-CAHPS Adult Visit Survey has acceptable psychometric properties at the individual level and practice site level. The analyses suggest that the survey items are measuring their intended concepts and yield reliable information.

Full-text (PDF)

Available from: Paul D Cleary, Jul 26, 2014
Psychometric Properties of the Consumer Assessment
of Healthcare Providers and Systems (CAHPS
s
) Clinician
and Group Adult Visit Survey
Naomi Dyer, PhD,* Joann S. Sorra, PhD,* Scott Alan Smith, PhD,* Paul D. Cleary, PhD,
w
and Ron D. Hays, PhD
zy
Background: The Consumer Assessment of Healthcare Providers
and Systems (CAHPS
s
) Clinician and Group Adult Visit Survey
enables patients to report their experiences with outpatient medical
offices.
Objective: To evaluate the factor structure and reliability of the
CAHPS Clinician and Group (CG-CAHPS) Adult Visit Survey.
Data Source: Data from 21,318 patients receiving care in 450
clinical practice sites collected from March 2010 to December 2010
were analyzed from the CG-CAHPS Database.
Research Design and Participants: Individual level and multilevel
confirmatory factor analyses were used to examine CAHPS survey
responses at the patient and practice site levels. We also estimated
internal consistency reliability and practice site level reliability.
Correlations among multi-item composites and correlations
between the composites and 2 global rating items were examined.
Measures: Scores on CG-CAHPS composites assessing Access to
Care, Doctor Communication, Courteous/Helpful Staff, and 2
global ratings of whether one would Recommend their Doctor, and
an Overall Doctor Rating.
Results: Analyses provide support for the hypothesized 3-factor
model assessing Access to Care, Doctor Communication, and
Courteous/Helpful Staff. In addition, the internal consistency reli-
abilities were Z 0.77 and practice site level reliabilities for sites
with >4 clinicians were Z 0.75. All composites were positively and
significantly correlated with the 2 global rating items, with Doctor
Communication having the strongest relationship with the global
ratings.
Conclusions: The CG-CAHPS Adult Visit Survey has acceptable
psychometric properties at the individual level and practice site
level. The analyses suggest that the survey items are measuring their
intended concepts and yield reliable information.
Key Words: CAHPS, consumer, survey, patient experience with
care, patient satisfaction, psychometrics, multilevel, confirmatory
factor analysis
(Med Care 2012;50: S28–S34)
T
he Consumer Assessment of Healthcare Providers and
Systems (CAHPS
s
) surveys were developed to elicit
reports from consumer s about their health care experiences.
The surveys cover topics such as the communication skills of
providers, helpfulness of staff, and access to care, which are
important to consumers and for which they are the best
source for this information. The surveys and accompanying
tools can be used by providers, health care organizations,
government agencies, and researchers to assess and improve
patient-centered care. Establishing the psychometric prop-
erties of CAHPS surveys is an integral step toward enabling
valid comparisons on patient experience across organizations
and over time.
1–5
The CAHPS Clinician and Group Survey (CG-CAHPS)
was developed to assess patient experiences with ambulatory
care. There are 3 versions of CG-CAHPS: (1) a 12-month
Survey that asks patients to report on their experiences over the
last 12 months; (2) an expanded 12-month Survey that includes
items to assess aspects of the Patient-Centered Medical Home;
and (3) a Visit Survey that primarily focuses on experiences
during a single visit. The Visit Survey includes questions about
doctor communication and office staff interactions at the pa-
tients most recent visit, and questions about the patient’s ac-
cess to care with their doctor over the last 12 months. The
survey also elicits an overall rating of the doctor from patients
and asks about their willingness to recommend their doctor’s
office to family and friends. The Visit Survey was designed to
collect feedback about a specific patient visit that providers can
use for monitoring and improving care.
In this paper, we evaluate the hypothesized factor
structure and reliability of the CG-CAHPS Adult Visit Sur-
vey using data submitted to the CG-CAHPS Database.
METHODS
Measures
The CG-CAHPS Adult Visit Survey contains 28
non-demographic items, of which 13 are used to create 3
From the *Westat, Rockville, MD; wYale School of Public Health, Yale
University, New Haven, CT; zRAND, Santa Monica, CA; and yUCLA
Department of Medicine, Los Angeles, CA.
Supported under contract (HHSA290200710024C) and by cooperative
agreements (U18HS016978 and U18HS016980) with the Agency for
Healthcare Research and Quality.
The authors declare no conflict of interest.
Reprints: Naomi Dyer, PhD, Westat, 1600 Research Blvd, Rockville, MD
20850. E-mail: naomidyer@westat.com.
Copyright
r
2012 by Lippincott Williams & Wilkins
ISSN: 0025-7079/12/5011-0S28
NEW CAHPS
s
SURVEYS
S28
|
www.lww-medicalcare.com Medical Care
Volume 50, Number 11 Suppl 3, November 2012
Advertisement:
composites that assess Access to Care (5 items), Doctor
Communication (6 items), and Courteous/Helpful Staff
(2 items). The survey als o includes 2 questions that ask re-
spondents (1) to rate their doctor; and (2) report if they
would recommend the doctor’s office to family and friends.
In addition, respondents are asked about their overall health,
age, sex, and education.
Access to Care Composite
The 5 Access to Care items ask patients about their
ability to get an appointment for urgent care as soon as
needed, get an appointment for a check -up or routine care as
soon as needed, get an answer to a phone question during
regular office hours on the same day, get an answer to a
phone question after hours as soon as needed, and if the wait
time to be seen was within 15 minutes of appointment time.
All questions in this composite have a reference period of
12 months and use a 4-point response scale (1 = Never,
2=Sometimes,3=Usually,4=Always). The Access to Care
composite uses a 12-month reference period unlike the other
items on the Visit Survey that ask about the most recent visit.
In field testing, results showed that the Access items using a
visit-based reference period did not achieve an acceptable
level of reliability. As a result, the Access items were
changed back to the 12-month reference period, leaving all
other items visit-specific.
Doctor Communication Composite
The 6 Doctor Communication items ask whether the
doctor explained things clearly, listened carefully, gave easy
to understand instructions, knew important medical history
about the patient, showed respect, and spent enough time
with the patient. These questions reference the most recent
visit and use a 3-point response scale (1 = Yes, definitely;
2=Yes, somewhat;3=No). The items in this composite were
recoded such that higher scores equal more positive re-
sponses (eg, Yes, definitely was recoded to 3; No was recoded
to 1).
Courteous/Helpful Staff Composite
The 2 Staffing items ask whether clerks and re-
ceptionists were helpful, and if they treated the patient with
courtesy and respect. These questions reference the most
recent visit and use a 3-point response scale (1 = Yes, defi-
nitely;2=Yes, somewhat;3=No). The items in this com-
posite were recoded such that higher scores equal more
positive responses (eg, Yes, definitely was recoded to 3; No
was reco ded to 1).
Overall Doctor Rating
This question asks the patient to rate the doctor on a
scale from 0 to 10, with 0 representing the worst doctor
possible and 10 representing the best doctor possibl e.
Recommend Doctor Rating
This question asks whether the patient would recom-
mend the doctor’s office to family and friends and uses a 3-
point response scale (1 = Yes, definitely;2=Yes, somewhat;
3=No). This item was recoded such that higher scores equal
more positive responses (eg, Yes, definitely was recoded to 3;
No was recoded to 1).
Analysis Dataset
The data was from the CG-CAHPS Database, con-
sisting of 103,442 respondents from 469 practice sites. The
Visit Survey includes a number of screener questions that
require a “yes” response before responding to a subsequent
question. For one of these questions, a majority of re-
spondents (93%) had not phoned their doctor after regular
office hours and therefore were instructed to skip the Access
to Care item Q12: “When you phoned this doctor’s office
after regular office hours, how often did you get an answer to
your medical question as soon as you needed it?” Because
there was such a high percentage of valid skips for this item,
it was dropped from further analyses. The remaining Access
to Care composite items had responses from between 46%
and 98% of the respondents. The 2 Courteous/Helpful Staff
items and 5 of the 6 Doctor Communication items were
answered by 99%. The Doctor Comm unication item (Q21)
about receiving easy to understand health care instructions
was answered by 84% of respondents.
To run a 3-factor psychometric model with items
loading onto their associated composites (Access, Doctor
Communication, and Courteous/Helpful Staff), we included
only nonmissing data for the items that make up the 3 CG-
CAHPS composites. The final analysis dataset therefore
consisted of 21,318 responses from 450 practice sites.
Sample
The data used for these analyses came from health
systems, medical offices, and survey vendors who volun-
tarily submitted CG-CAHPS survey data collected from
March 2010 to December 2010 to the CAHPS Database. All
of the 450 practice sites included in the analysis dataset
administered mail surveys. Most of the practice sites speci-
alized in Family Practice and/or Internal Medicine (89%).
Over two thirds of the practice sites were owned by a hos-
pital or integrated delivery system (69%). Most respondents
were female (67%) and a majority were 45 years or older
(81%).
Analyses
Descriptive statistics for the survey items and Spear-
man rank-order correlations with their associated composites
and the global rating items were computed. In addition, we
performed confirmatory factor analyses using Mplus Version
6.12, as described below. Finally, we estimated internal
consistency reliability and physician group-level reliability
(see below).
Individual-level Confirmatory Factor Analysis
We conducted individual-level confirmatory factor
analysis on the proposed 3-factor model, with maximum
likelihood estimation, at first ignoring the nesting of re-
spondents within practice sites. To assess the appropri ateness
of the resulting structure, we examined factor loadings with
the criterion that they should be Z 0.40.
6
We present
standard overall model fit statistics: the w
2
, comparative fit
Medical Care
Volume 50, Number 11 Suppl 3, November 2012 Psychometric Properties of CG-CAHPS Adult Visit Survey
r
2012 Lippincott Williams & Wilkins www.lww-medicalcare.com
|
S29
index (CFI), the root mean square error of approximation
(RMSEA), and the standardized root mean square residua l
(SRMR).
Given the large sample size of our dataset, we pri-
marily relied on the CFI, RMSEA, and SRMR as indices of
model fit because the w
2
is influenced by sample size such
that the lar ger the sample size the more likely it is that the w
2
will be significant (which indicates lack of model fit).
7,8
The
CFI compares the existing model fit with a null model that
assumes the items in the model are uncorrelated. The factor
structure is determined to adequately fit the data if the CFI is
at least 0.95.
9
The RMSEA examines the residuals of the
model; an RMSEA of r0.06 is indicative of good fit.
9
The
SRMR is the standardized difference between the observed
and predicted covariances from the model. A value of 0 for
the SRMR indicates perfect fit, but a value <0.08 is con-
sidered good fit.
9
Multilevel Analyses
When respondent data are nested within practice sites,
multilevel modeling is generally more appropriate because it
accounts for the nested nature of the data. We performed a
number of steps in association with the multilevel analyses.
Intraclass Correlations (ICCs) and Design Effects
First, we examined ICCs and design effects to de-
termine if the data were truly nested and therefore multilevel
analyses would be necessary.
10
ICCs > 0.05 indicate that
the multilevel structure of the data needs to be taken into
consideration; ICCs < 0.05 signify that the consequences of
not using multilevel analyses are minimal.
11
We also ex-
amined design effects, as ICC s are affected when there are
few groups comprised of many individuals or many groups
comprised of few individuals, as is the case for our dataset.
Design effects take into consideration the group sample size
(Design effect = 1+[Average within group sample size–1]
ICC). A design effect of Z 2.0 implies that group member-
ship is associated with responses of the indivi duals and
therefore multilevel modeling should be conducted to ac-
count for the multilevel nature of the data.
12
Multilevel Confirmatory Factor Analyses (MCFA)
Similar to the individual-level confirmatory factor
analyses, a 3-factor model was examined, taking into con-
sideration the nested nature of the data. We evaluated the
item factor loadings with the same rule as the individual
level confirmatory factor analyses—that factor loadings
should be Z 0.40. With multilevel models, 2 sets of factor
loadings are provided: between-practice sites and within-
practice sites, which coincide with the nested nature of the
data. The between factor loadings are based on the between-
practice site covariance matrix, whereas the within factor
loadings use the within-level or respondent-level covariance
matrix. We agai n present overall mode l fit indices using
standard fit statistics: the w
2
, CFI, RMSEA, and SRMR, with
the same criteria as at the individual level.
Reliability
Cronbach coefficient a, an estimate of reliability, was
calculated for each composite to assess the extent to which
respondents consistently answered the items, with a reli-
ability of at least 0.70 considered acceptable.
13
We examined practice site reliability by practice site
size (ie, the number of clinicians per site) because practices
of different sizes need different numbers of patient surve ys
to reach acceptable levels of reliability on the measures.
We calculated practice site reliability using the following
formula:
Reliability
g
¼
S
B
S
B
þ
S
W
N
g
;
where S
B
refers to the between-group variance; S
W
refers to
the within-group variance, and N
g
is the sample size for
practice site g.
14
Average reliability estimates were calculated for the 3
composites and 2 global rating items for 6 practice size
categories: (1) 1 clinician; (2) 2–3 clinicians; (3) 4–9 clini-
cians; (4) 10–13 clinicians; (5) 14–19 clinicians; and (6)
Z 20 clinicians. A variety of different size categories were
considered and other splits are possible but this set of cate-
gories was chosen based on variance in reliability and patient
sample sizes available in our dataset. Similar to internal
consistency reliability, values of at least 0.70 are considered
acceptable for practice site comparisons.
13
Correlations Among Composites and Global Ratings
Relationships among the composites and global ratings
at the individual and practice site levels were also examined
using Spearman rank- order correlations. Although the com-
posites should be correlated as they all measure aspects of
patient experience, very high intercorrelations indicate that
the composites may not be unique enough to be considered
separate measures. In general, composi te intercorrelatio ns
should be <0.80 for the composites to be considered
unique.
15
We hypothesized that the composites would be
positively related to the global rating items.
RESULTS
Descriptive Statistics
The means, SDs, top box scores, and correlations for
the survey items are provided in Table 1. Consistent with
other patient experience data, CG-CAHPS ratings of care
tend to be very positive (negatively skewed)—that is, con-
sumers tend to report positive experiences with health care in
the United States.
16
The item-to-composite correlations (corrected for item
overlap with the composite total) ranged from 0.40 (Q13
with Access to Care) to 0.71 (Q28 and Q29 with Courteous/
Helpful Staff). The correlations between the composite items
and the global rating items ranged from 0.18 (Q29. Cour-
teous/Helpful Staff with Overall Doctor Rating) to 0.53
(Q19. Doc tor Communication with Recommend Doctor).
Dyer et al Medical Care
Volume 50, Number 11 Suppl 3, November 2012
S30
|
www.lww-medicalcare.com
r
2012 Lippincott Williams & Wilkins
Individual-level Confirmatory Factor Analysis
Table 2 shows that all items within the composites
had factor loadings greater than the 0.40 criterion with an
average loading of 0.68 for Access to Care, 0.76 for Doctor
Communication, and 0.86 for Courteous/Helpful Staff. The
overall model fit indices are shown in Table 3. As expected,
the w
2
test was statistically significant (P < 0.01) given the
large sample size. The CFI was 0.97, >0.95 criterion for good
model fit. The RMSEA was 0.05, <0.06 criterion, indicating
good model fit. The SRMR was 0.04, <0.08 criterion, again
signifying good model fit. Overall, the individual-level factor
analysis results provided initial support for the 3 composites
and justification for aggregating the items into their asso-
ciated com posites.
Multilevel Factor Analyses
ICCs and Design Effects
As shown in Table 2, the item ICCs for Access to Care
were all greater than the 0.05 criterion; with an average of
0.08, ranging from 0.07 to 0.11. This finding indicates that
between 7% and 11% of the variance may be attributed to
practice site membership and establishes the need for mul-
tilevel analyses. For Doctor Communication and Courteous/
Helpful Staff, all the item ICC values were r0.05 criterion
indicating very little variability across practice sites (average
of 0.02, ranging from 0.01 to 0.05). However, when exam-
ining design effects, both Courteous/Helpful Staff items and
one of the Doctor Communication items had values ex-
ceeding the 2.00 criterion indicating the nested nature of the
data for these items. Overall, these statistics confirmed that,
in general, responses within practice sites were more similar
than would be expected by chance; therefore the clustered
nature of the data should be taken into account when ex-
amining their factor structure.
MCFA
All factor loadings estimated with the multilevel
models were greater than the 0.40 criterion (Table 2). The
between-practice site factor loadings ranged from 0.59 to
0.99 and the within-practice site factor loadings ranged from
0.45 to 0.99. The w
2
test (Table 3) was significant (P < 0.01)
as expected, but CFI was 0.97, >0.95 criterion. In addition,
the RMSEA was 0.03, <0.06 criterion, indicating good fit.
The within-practice site SRMR was 0.05, <0.08 criterion that
indicated good fit; however, the between-practice site SRMR
was slight ly above the cutoff at 0.10.
Reliability
All composites had acceptable (Z 0.70) individual
level internal consistency reliability estimates, ranging
from 0.77 to 0.89 (Table 4). Practice site level reliability
was examined across the composites and global rating items
by practice site size categories (1 clinician to Z 20
clinicians, Table 5). The practice site reliability estimates
were acceptable for all sites with at least 4 clinicians. For
sites with 1 clinician, only Access to Care had reliability
>0.70. The remaining reliabilities for practice sites with 1
clinician ranged from 0.40 (Courteous/Helpful Staff) to 0.69
(Overall Rating Item). For sites with 2–3 clinicians,
TABLE 1. Descriptive Statistics for CG-CAHPS Adult Visit Items Wth Associated Composite and Global Ratings
Spearman Rank-Order Correlations
Composites, Items, and Global Ratings Top Box (%) Mean SD Associated Composite Recommend Doctor Overall Rating
Access to Care Composite
Q6 Got appointment when needed care right away 64 3.52 0.73 0.59 0.31 0.33
Q8 Got appointment for check-up or routine care 68 3.60 0.66 0.59 0.30 0.31
Q10 Got answer to medical question same day 60 3.46 0.76 0.50 0.30 0.33
Q13 Visit started <15 min of appointment time 40 3.14 0.87 0.40 0.23 0.26
Doctor Communication Composite
Q18 Did doctor explain things in a way easy to understand 92 2.91 0.31 0.55 0.47 0.38
Q19 Did doctor listen carefully 93 2.93 0.29 0.57 0.53 0.39
Q21 Did doctor give easy to understand instructions 91 2.89 0.35 0.58 0.50 0.41
Q22 Did doctor know important medical history 90 2.89 0.36 0.49 0.45 0.41
Q23 Did doctor show respect 95 2.94 0.27 0.51 0.52 0.36
Q24 Did doctor spend enough time 91 2.89 0.35 0.53 0.48 0.40
Courteous/Helpful Staff Composite
Q28 Clerks and receptionists helpful 88 2.86 0.38 0.71 0.28 0.22
Q29 Clerks and receptionists treat with courtesy and respect 92 2.91 0.31 0.71 0.26 0.18
Single Item Global Ratings
Q25 Overall doctor Rating 82 9.27 1.33
Q26 Recommend doctor 90 2.88 0.39
Responses were from 450 practice stes and 21,318 respondents. The Spearman correlation with the associated composite is the item-to-composite corrected correlation.
CG-CAHPS indicates Consumer Assessment of Healthcare Providers and Systems Clinician and Group.
Medical Care
Volume 50, Number 11 Suppl 3, November 2012 Psychometric Properties of CG-CAHPS Adult Visit Survey
r
2012 Lippincott Williams & Wilkins www.lww-medicalcare.com
|
S31
both Access to Care and Courteous and Helpful Staff had
reliability estimates >0.70. The remaining reliabilities ranged
from 0.58 (Recommend Doctor item) to 0.66 (Overall Rating
item). The average number of respondents in 1 clinician and
2–3 clinician o ffices was <100, indicating that for these
smaller sites it is necessary to have more respondents per
practice site to increase reliability to acceptable levels.
Spearman Correlations Among the Composites
and Global Ratings
All Spearman rank-order composite correlations were
statistically significant (P < 0.01), and none of the correlations
exceeded the 0.80 criterion signaling potential multi-
collinearity (Table 4). The average individual-level correlation
among the composites was 0.30 (range: r =0.250.35). The
average practice site level correlation among the composites
was 0.48 (range: r = 0.41–0.57). The lowest correlations at the
individual and practice site levels were between Doctor
Communication and Courteous/Helpful Staff (0.25 for in-
dividual and 0.41 for practice site level, respectively). The
highest correlation at the individual level was between Access
to Care and Doctor Communication (r = 0.35). The highest
correlation at the practice site level was between Access to
Care and Courteous/Helpful Staff (r =0.57).
TABLE 2. Confirmatory Factor Analysis Results of the CG-CAHPS Adult Visit Survey
Individual-level CFA Practice Site Level Multilevel CFA
Composites and Items Factor Loading ICC Design Effect Within Factor Loading Between Factor Loading
Access to Care Composite
Q6 Got appointment when needed care right away 0.82 0.08 4.66 0.81 0.98
Q8 Got appointment for check-up or routine care 0.81 0.07 4.06 0.81 0.98
Q10 Got answer to medical question same day 0.62 0.07 4.15 0.60 0.91
Q13 Visits started <15 min of appointment time 0.46 0.11 6.24 0.45 0.59
Doctor Communication Composite
Q18 Did doctor explain things in a way easy to understand 0.75 0.01 1.51 0.75 0.99
Q19 Did doctor listen carefully 0.85 0.02 1.70 0.85 0.98
Q21 Did doctor give easy to understand instructions 0.77 0.01 1.60 0.77 0.96
Q22 Did doctor know important medical history 0.65 0.02 2.02 0.65 0.94
Q23 Did doctor show respect 0.81 0.02 1.70 0.81 0.99
Q24 Did doctor spend enough time 0.73 0.02 1.93 0.73 0.91
Courteous/Helpful Staff Composite
Q28 Clerks and receptionists helpful 0.91 0.05 3.27 0.74 0.99
Q29 Clerks and receptionists treat with courtesy and respect 0.81 0.04 2.95 0.99 0.82
CFA indicates Confirmatory Factor Analysis; CG-CAHPS, Consumer Assessment of Healthcare Providers and Systems Clinician and Group; ICC, intraclass correlation.
TABLE 3. Fit Indices for 3-factor Confirmatory Factor Analysis
Models of the CG-CAHPS Adult Visit Survey
Fit
Statistics
Individual-level
CFA
Practice Site Level Multilevel
CFA
w
2
3129.6 2551.1
df 51 105
CFI 0.97 0.97
RMSEA 0.05 0.03
SRMR 0.04
Within 0.05
Between 0.10
w
2
statistics are significant, P < 0.01.
CFA indicates Confirmatory Factor Analysis; CG-CAHPS, Consumer Assessment
of Healthcare Providers and Systems Clinician and Group; RMSEA, root mean square
error of approximation; SRMR, standardized root mean square residual.
TABLE 4. Individual-level Reliability and Individual and
Practice Site Level Spearman Rank-Order Correlations Among
CG-CAHPS Adult Visit Composites and Global Ratings
Composites and Global Rating
Items
Composites and Global Ratings 1 2 3 4 5
Access to Care Composite (0.77) 0.45 0.57 0.42 0.52
Doctor Communication Composite 0.35 (0.89) 0.41 0.75 0.76
Courteous/Helpful Staff Composite 0.29 0.25 (0.85) 0.34 0.43
Overall Doctor Rating Item 0.39 0.52 0.22 0.76
Recommend Doctor Item 0.34 0.52 0.29 0.47
Individual-level Cronbach a shown in parentheses ( ). The top right part of matrix
displays practice site level correlations. Lower left, bolded part of matrix displays
individual-level correlations. All Spearman correlations are statistically significant,
P < 0.01.
CG-CAHPS indicates Consumer Assessment of Healthcare Providers and Systems
Clinician and Group.
Dyer et al Medical Care
Volume 50, Number 11 Suppl 3, November 2012
S32
|
www.lww-medicalcare.com
r
2012 Lippincott Williams & Wilkins
The Spearman correlations between the composites
and the 2 global rating items were all statistically significant
(P < 0.01). For the Overall Doctor Rating item, the average
individual level correlation with the composites was 0.38
(range: r = 0.22–0.52) and the average practice site level
correlation was 0.50 (range: r = 0.34–0.75 ). For the Recom-
mend Doctor item, the average individual-level correlation
with the composites was 0.38 (range: r = 0.29–0.52), whereas
the average practice site level correlation was 0.57 (range:
r = 0.43–0.76). The highest correlation with the global rating
items was with the Doctor Communication composite and
the Recommend Doctor item (0.52 at the individual level and
0.76 at the practice site level). Finally, the Spearman cor-
relations between the 2 global ratings were 0.47 and 0.76 at
the individual and practice site levels, respectively.
DISCUSSION
The CG-CAH PS Adult Visit survey is a publicly
available, standardized tool to measure patients’ experiences
with outpatient medical offices. Demonstrating the psycho-
metric properties of the survey is an important step for fur-
thering its use. Overall, both the individual level and
multilevel CFA results provided support for the survey’s 3
composites (Access to Care, Doctor Communication, Courteous/
Helpful Staff) and 2 global rating items (Overall Doctor
Rating, Recommend Doctor).
This study of a large number of practice sites and a
large sample of patients provides support that the CG-
CAHPS composites have acceptable individual-level intern al
consistency reliability and practice site level reliability.
Practice-level reliability is important because the survey is
intended to provide information at the practice level, for
public reporting of patient experience data, and to enable
confidence in comparisons of data across sites. In our dataset,
we found acceptable practice site level reliability for sites
with at least 4 clinicians. The reliability stays relatively
the same, and >0.70, across sites with 4–Z 20 clinicians
(Table 5). Given that site-level reliabili ty is a function of
sample size, and the average sample size for practice sites
with >4 clinicians was far less than those with Z 4, these
practice sites could achieve adequate site-level reliability by
requiring responses from more respondents than were
available in our dataset.
The CG-CAHPS survey, in providing the patient’s
perspective, is critical for achieving the Institute of Medi-
cine’s aim of patient-centered care and for improving quality
of care in outpatient medical offices. Num erous studies have
linked patient experience data in various settings to better
clinical outcomes, patient adherence to medications, patient
retention in physicians’ practices, and lower medical mal-
practice risk.
17
It is therefore important to have reliable and
valid measures for assessing patient experience.
The associations between the composites and global
rating items provide support for the construct validity of the
CG-CAHPS measures. Doctor Communication had the
strongest relationship with the global ratings, which is con-
sistent with earlier studies that have shown Doctor Com-
munication to be a key driver of patients’ overall ratings of
their doctor and their willingness to recommend their doc-
tor’s office.
1,2,4
The Courteous/Helpful Staf f composite had
the weakest relationships with the global ratings suggesting
that staff play less of a role in patients’ global assessments of
their doctors.
It should be noted that while there were a large number
of practice sites included in our dataset, they are not stat-
istically representative of all medical offices in the United
States because the data came from sites and states that vol-
untarily submitted their data to the CAHPS Database. Nev-
ertheless, the analyses presented here represent one of the
largest samples of medical offices studied and provide
compelling support for the reliability, factor struct ure, and
construct validity of the CG-CAHPS Adult Visit survey.
Future research is needed to assess the associations of CG-
CAHPS survey responses with clinical process measures and
health outcome s.
ACKNOWLEDGMENT
The authors thank Dale Shaller for facilitating access
to the CAHPS Database.
REFERENCES
1. Solomon LS, Hays RD, Zaslavsky AM, et al. Psychometric properties of
a group-level Consumer Assessment of Health Plans Study (CAHPS)
instrument. Med Care. 2005;43:53–60.
2. Hargraves JL, Hays RD, Cleary PD. Psychometric Properties of the
Consumer Assessment of Health Plans Study (CAHPS) 2.0 Adult Core
Survey. Health Serv Res. 2003;38(part I):1509–1527.
3. Hays RD, Brown J, Brown L, et al. Classical test theory and item
response theory analyses of multi-item scales assessing parents’
TABLE 5. Average Practice Site Level Reliabilities by Size for CG-CAHPS Adult Visit Survey
Practice Site Size
Composites/Items 1 Clinician 2–3 Clinicians 4–9 Clinicians 10–13 Clinicians 14–19 Clinicians Z20 Clinicians
No. practice sites 36 84 211 53 49 18
Access to Care Composite 0.77 0.70 0.83 0.91 0.89 0.81
Doctor Communication Composite 0.62 0.62 0.78 0.79 0.81 0.79
Courteous/Helpful Staff Composite 0.40 0.73 0.84 0.89 0.91 0.76
Overall Doctor Rating Item 0.69 0.66 0.81 0.84 0.83 0.82
Recommend Doctor Item 0.66 0.58 0.75 0.82 0.81 0.83
Average respondent, N 48 90 200 296 298 602
Practice site level reliabilities were calculated on the larger dataset consisting of 469 practice sites excluding 8 sites that were unable to be categorized by size.
CG-CAHPS indicates Consumer Assessment of Healthcar e Providers and Systems Clinician and Group.
Medical Care
Volume 50, Number 11 Suppl 3, November 2012 Psychometric Properties of CG-CAHPS Adult Visit Survey
r
2012 Lippincott Williams & Wilkins www.lww-medicalcare.com
|
S33
perceptions of their children’s dental care. Med Care. 2006;44(suppl
3):S60–S68.
4. Hays RD, Shaul JA, Williams V, et al. Psychometric properties of the
CAHPSt 1.0 survey measures. Med Care. 1999;37(suppl 3):MS22–
MS31.
5. O’Malley AJ, Zaslavsky AM, Hays RD, et al. Exploratory factor
analyses of the CAHPSs Hospital Pilot Survey Responses across and
within Medical, Surgical, and Obstetric Services. Health Serv Res.
2005;40(part II):2078–2095.
6. Peterson RAA. Meta-analysis of variance accounted for and factor
loadings in exploratory factor analysis. Mark Lett. 2000;11:261–275.
7. Brown TA. Confirmatory Factor Analysis for Applied Research. New
York: The Guilford Press; 2006.
8. Kenny DA. Measuring Model Fit. 2011. September 4, 2011. Available
at: http://davidakenny.net/cm/fit.htm. Accessed July 18, 2012.
9. Hu LT, Bentler PM. Cutoff criteria for fit indices in covariance structure
analysis: conventional criteria versus new alternatives. Struct Equ
Modeling. 1999;6:1–55.
10. Muthen BO, Muthen LK. Mplus: User’s Guide. Los Angeles, CA:
Muthen & Muthen; 1998.
11. Julian M. The consequences of ignoring multilevel data structures in
nonhierarchical covariance modeling. Struct Eq u Modeling. 2001;8 :325–352.
12. Muthen BO, Satorra A. Complex sample data in structural equation
modeling. In: Marsden P, ed. Sociological Methodology. San Francisco:
Jossey-Bass; 1995:267–316.
13. Nunnally JC, Bernstein IH. Psychometric Theory. New York: McGraw
Hill; 1994.
14. Raudenbush SW, Bryk AS. Hierarchical Linear Models. 2nd ed.
Thousand Oaks, CA: Sage; 2002.
15. O’Brien RM. A caution regarding rules of thumb for variance inflation
factors. Qual Quant. 2007;41:673–690.
16. Nelson. EC, Gentry MA, Mook KH, et al. How many patients are
needed to provide reliable evaluations of individual clinicians? Med
Care. 2004;42:259–266.
17. Browne K, Roseman D, Shaller D, et al. Measuring patient experience
as a strategy for improving primary care. Health Aff. 2010;5:921–925.
Dyer et al Medical Care
Volume 50, Number 11 Suppl 3, November 2012
S34
|
www.lww-medicalcare.com
r
2012 Lippincott Williams & Wilkins
    • "In addition to sociodemographic and self-reported health status variables, patients were asked to rate three somewhat different dimensions of visit satisfaction: [29]. One item (the doctor gave easy to understand instructions) was inadvertently omitted from the scale. "
    [Show abstract] [Hide abstract] ABSTRACT: Objective: Cancer patients use complementary and alternative medicine (CAM), but do not routinely talk about it with their clinicians. This study describes CAM discussions in oncology visits, the communication patterns that facilitate these discussions and their association with visit satisfaction. Methods: 327 patients (58% female; average age 61) and 37 clinicians were recorded during an oncology visit and completed post-visit questionnaires. All CAM discussions were tagged and the Roter Interaction Analysis System (RIAS) was used to code visit dialogue. Results: CAM was discussed in 36 of 327 visits; discussions were brief (<one minute), the majority patient initiated (65%) and more common for patients in early stages of cancer care. Longer visits (35 vs 29min; p<0.05), greater patient engagement in visit dialogue, lowered clinician verbal dominance and a more patient-centered pattern of visit communication were significantly related to visits with CAM discussions (all p values<0.01). Both patient and clinician visit satisfaction was higher with CAM discussion (p<0.05). Conclusions: CAM discussions do not occur at random; they take place in visits characterized by patient-centered communication and are associated with higher visit satisfaction. Practice implications: CAM discussions are perceived positively by both patients and clinicians and are facilitated by patient-centered visit communication.
    Full-text · Article · Jun 2016
    • "We casemix adjusted each score for patient characteristics, as is necessary when comparing CG- CAHPS Ò measures across organizations or conducting individual physicians comparisons (Zaslavsky et al. 2001) because some patient factors that are associated with CG-CAHPS Ò scores are not amenable to intervention by health delivery organizations or physicians. The CG-CAHPS Ò instructions (Dyer et al. 2012) for analyzing the data recommend that users calculate case-mix adjusted scores for each of the survey composite measures (communication , access, office staff interactions) using patient age (18–24, 25–34, 35–44, 45–54, 55–64, 65–74, 75–84, and 85 years or older), education (\ 8th grade, some high school, high school graduate, some college, college graduate, and graduate school), and selfreported health (excellent, very good, good, fair, and poor). We used these same case-mix adjusters. "
    [Show abstract] [Hide abstract] ABSTRACT: Modern psychometric methods for scoring the Clinician & Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS®) instrument can improve the precision of patient scores. The extent to which these methods can improve the reliable estimation and comparison of individual physician performance, however, remains unclear. Using CG-CAHPS® data from 12,244 unique patients of 448 primary care physicians in southern California, four methods were used to calculate composite scores: (1) standard scoring, (2) a single factor confirmatory factor analysis model, (3) a bifactor model, and (4) a correlated factor model. We extracted factor scores for physicians from each model and adjusted the scores for respondent characteristics, including age, education, self-rated physical health, and race/ethnicity. Physician-level reliability and physician rankings were examined across the four methods. The bifactor and correlated factor models achieved the best fit for the core CG-CAHPS® questions from the three core composite measures. Compared to standard adjusted scoring, the bifactor model scores resulted in a 25 % reduction in required sample sizes per physician. The correlation of physician rankings between scoring methods ranged from 0.58 to 0.86. The discordance of physician rankings across scoring methods was most pronounced in the middle of the performance distribution. Using modern psychometric methods to score physician performance on the core CG-CAHPS® questions may improve the reliability of physician performance estimates on patient experience measures, thereby reducing the required respondent sample sizes per physician compared to standard scoring. To assess the predictive validity of the CG-CAHPS® scores generated by modern psychometric methods, future research should examine the relative association of different scoring methods and important patient-centered outcomes of care.
    Full-text · Article · Dec 2013
    • "Health Plan survey (Dyer, Sorra, Smith, Cleary, & Hays, 2012) asking: "
    [Show abstract] [Hide abstract] ABSTRACT: Background: There is evidence that Black patients may experience stereotype threat--apprehension about being negatively stereotyped--in healthcare settings, which might adversely affect their behavior in clinical encounters. Recent studies conducted outside of healthcare have shown that a brief self-affirmation intervention, in which individuals are asked to focus on and affirm their valued characteristics and sources of personal pride, can reduce the negative effects of stereotype threat on academic performance and on interpersonal communication. Methods: This randomised controlled trial examined whether a self-affirmation (SA) intervention would decrease the negative effects of stereotype threat (negative mood, lower state self-esteem, greater perceptions of racial discrimination) and increase communication self-efficacy among Black primary care patients. Self-affirmation was induced by having patients complete a 32-item values affirmation questionnaire. Results: Patients in the SA condition had lower levels of performance self-esteem and social self-esteem than patients in the control. There were no differences between the SA and the control groups on negative mood, communication self-efficacy, and perceptions of discrimination. Conclusions: Our SA intervention lowered state self-esteem among Black patients. Future research is needed to determine the type of SA task that is most effective for this population.
    Article · Oct 2013
Show more