Content uploaded by Hugo ten Cate
Author content
All content in this area was uploaded by Hugo ten Cate
Content may be subject to copyright.
Performance of 4 Clinical Decision Rules in the Diagnostic
Management of Acute Pulmonary Embolism
A Prospective Cohort Study
Rene´e A. Douma, MD; Inge C.M. Mos, MD; Petra M.G. Erkens, MSc; Tessa A.C. Nizet, MD; Marc F. Durian, MD; Marcel M. Hovens, MD;
Anja A. van Houten, MD; Herman M.A. Hofstee, MD; Frederikus A. Klok, MD; Hugo ten Cate, MD; Erik F. Ullmann, MD;
Harry R. Bu¨ller, MD; Pieter W. Kamphuisen, MD; and Menno V. Huisman, MD, for the Prometheus Study Group*
Background: Several clinical decision rules (CDRs) are available to
exclude acute pulmonary embolism (PE), but they have not been
directly compared.
Objective: To directly compare the performance of 4 CDRs
(Wells rule, revised Geneva score, simplified Wells rule, and
simplified revised Geneva score) in combination with D-dimer
testing to exclude PE.
Design: Prospective cohort study.
Setting: 7 hospitals in the Netherlands.
Patients: 807 consecutive patients with suspected acute PE.
Intervention: The clinical probability of PE was assessed by using a
computer program that calculated all CDRs and indicated the next
diagnostic step. Results of the CDRs and D-dimer tests guided
clinical care.
Measurements: Results of the CDRs were compared with the
prevalence of PE identified by computed tomography or venous
thromboembolism at 3-month follow-up.
Results: Prevalence of PE was 23%. The proportion of patients
categorized as PE-unlikely ranged from 62% (simplified Wells rule)
to 72% (Wells rule). Combined with a normal D-dimer result, the
CDRs excluded PE in 22% to 24% of patients. The total failure
rates of the CDR and D-dimer combinations were similar (1 failure,
0.5% to 0.6% [upper-limit 95% CI, 2.9% to 3.1%]). Even though
30% of patients had discordant CDR outcomes, PE was not de-
tected in any patient with discordant CDRs and a normal D-dimer
result.
Limitation: Management was based on a combination of decision
rules and D-dimer testing rather than only 1 CDR combined with
D-dimer testing.
Conclusion: All 4 CDRs show similar performance for exclusion of
acute PE in combination with a normal D-dimer result. This pro-
spective validation indicates that the simplified scores may be used
in clinical practice.
Primary Funding Source: Academic Medical Center, VU University
Medical Center, Rijnstate Hospital, Leiden University Medical Cen-
ter, Maastricht University Medical Center, Erasmus Medical Center,
and Maasstad Hospital.
Ann Intern Med. 2011;154:709-718. www.annals.org
For author affiliations, see end of text.
* For a list of members of the Prometheus Study Group, see Appendix 1
(available at www.annals.org).
The introduction of standardized clinical decision rules
(CDRs) to determine the clinical probability of pulmo-
nary embolism (PE) has improved the diagnostic work-up
of patients with suspected PE. A CDR result of “PE-
unlikely” in combination with a normal D-dimer result
can exclude PE in a large proportion of patients who
present for evaluation (20% to 40%), without the need
for additional imaging with computed tomographic pul-
monary angiography (CTPA) or ventilation–perfusion
(V
˙/Q
˙) scintigraphy, both of which involve radiation and
intravenous contrast or radioisotopes. In these patients, an-
ticoagulant drugs can be safely withheld (1–4).
Several CDRs that incorporate information from med-
ical history and physical examination have been developed
and validated. In addition to 6 objective variables, the
Wells rule contains 1 subjective variable: The physician
should consider the possibility of a diagnosis other than PE
for the patient’s symptoms (Table 1) (5). In contrast, the
more recent revised Geneva score comprises 8 objective
clinical variables (6). Both scores assign different
weights to the variables, meaning that depending on the
variable, 1, 1.5, 2, 3, 4, or 5 points need to be assigned
(Table 1). Because miscalculations can occur, the scores
have been simplified (Table 1) (7, 8).
Until now, the simplified Wells rule and the simplified
revised Geneva score have not been validated prospectively.
Also, although some of the scores have been retrospectively
or prospectively compared with each other (9–12), the 4
scores have never been directly compared for their perfor-
mance in excluding PE in combination with a normal
D-dimer result. Therefore, we did a prospective, multi-
center study on clinical accuracy to assess and directly com-
See also:
Print
Editors’ Notes .............................710
Web-Only
Appendixes
Appendix Tables
Appendix Figures
CME quiz
Conversion of graphics into slides
Annals of Internal Medicine Original Research
© 2011 American College of Physicians 709
pare the performance of 4 CDRs (Wells rule, revised Ge-
neva score, simplified Wells rule, and simplified revised
Geneva score) in excluding PE in combination with a nor-
mal D-dimer result by using a computer-based program to
calculate the CDR scores.
METHODS
Patients
We performed a prospective, multicenter cohort study on
clinical accuracy of 4 CDRs in consecutive patients with a
suspected first episode of acute PE. The study population con-
sisted of consecutive outpatients and inpatients in whom a
first acute PE was clinically suspected. Clinically suspected
acute PE was defined as sudden onset of dyspnea, deteriora-
tion of existing dyspnea, or sudden onset of pleuritic chest
pain. Patients were included from 7 participating academic or
nonacademic hospitals in the Netherlands.
Exclusion criteria were age younger than 18 years, life
expectancy less than 3 months, treatment with therapeutic-
dose low-molecular-weight heparin or unfractionated heparin
that was initiated 24 hours or more before eligibility assess-
ment, treatment with vitamin K antagonists, previous diagno-
sis of PE, contraindication to helical computed tomography
(CT) because of allergy to intravenous iodinated contrast or
renal insufficiency (creatinine clearance ⬍30 mL/min per
1.73 m
2
by using the Cockroft–Gault formula), pregnancy,
and inability to return for follow-up. Institutional review
boards of all participating hospitals approved the study proto-
col, and written informed consent was obtained from in-
cluded patients.
Study Flow
Included patients had a sequential work-up involving
clinical probability assessment, D-dimer testing, and CT.
Treating physicians assessed the items of 4 CDRs in all
patients (Table 1). In addition, a high-sensitivity quantita-
tive D-dimer test was done (VIDAS D-dimer assay, bio-
Me´rieux, Marcy-l’E
´toile, France; Tina-quant assay, Roche
Diagnostica, Mannheim, Germany; STA Liatest D-Di, Di-
agnostica Stago, Asnie`res-sur-Seine, France; or Innovance
D-dimer, Siemens, Marburg, Germany) for all included pa-
tients, regardless of CDR results. The type of D-dimer assay
that was used depended on the local practice. Pulmonary
embolism was considered unlikely if the Wells rule score
was 4 points or less, the simplified Wells rule score was 1
point or less (7, 10), and the simplified revised Geneva
score was 2 points or less (Table 1) (8). The revised Ge-
neva score was formerly available only in a 3-category
scheme, but it was recently made into a 2-category scheme,
similar to the other scores. This was done by using a be-
forehand calculation of the optimal cutoff in an existing
cohort of patients with suspected PE (8) for whom the
revised Geneva score variables were available to calculate
the score. The optimal cutoff point was determined by
calculating the area under the receiver-operating character-
istic curve, and the proportions of patients classified in the
likely and unlikely categories were calculated (Appendix 2,
available at www.annals.org). Patients were considered PE-
unlikely if they scored 5 points or less (Table 1). A score
greater than the respective cutoff indicated a classification
of PE-likely for any of the 4 CDRs.
Table 1. Clinical Decision Rules
Clinical Decision Rule Points
Original
Version
Simplified
Version
Wells rule
Previous PE or DVT 1.5 1
Heart rate ⬎100 beats/min 1.5 1
Surgery or immobilization within 4 wk 1.5 1
Hemoptysis 1 1
Active cancer 1 1
Clinical signs of DVT 3 1
Alternative diagnosis less likely than PE 3 1
Clinical probability
PE unlikely ⱕ4ⱕ1
PE likely ⬎4⬎1
Revised Geneva score
Previous DVT or PE 3 1
Heart rate
75–94 beats/min 3 1
ⱖ95 beats/min 5 2
Surgery or fracture within 1 mo 2 1
Hemoptysis 2 1
Active cancer 2 1
Unilateral lower limb pain 3 1
Pain on lower limb deep venous
palpation and unilateral edema
41
Age ⬎65 y 1 1
Clinical probability
PE unlikely ⱕ5ⱕ2
PE likely ⬎5⬎2
DVT ⫽deep venous thrombosis; PE ⫽pulmonary embolism.
Context
Several clinical decision rules (CDRs) are available to evalu-
ate patients with possible pulmonary embolism (PE). It is
not known which CDR, if any, is best to use.
Contribution
In this multicenter, prospective study, 4 CDRs were used
to determine whether PE was likely or unlikely, combined
with the results of D-dimer testing, and did equally well at
excluding PE or indicating the need for further testing. The
CDRs were the Wells rule, the revised Geneva score, the
simplified Wells rule, and the simplified revised Geneva
score.
Implication
Provided that these 4 CDRs are used correctly, clinicians
can confidently choose them according to personal or in-
stitutional preferences to assist in the evaluation of possi-
ble PE.
—The Editors
Original Research Clinical Decision Rules in the Management of Acute Pulmonary Embolism
710 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 www.annals.org
Clinical care was guided by the results of the CDRs and
D-dimer testing (Figure 1). When PE was considered unlikely
according to all 4 CDRs in combination with a normal
D-dimer result (cutoff ⬍500
cg/L), PE was excluded. In all
remaining patients (those considered PE-likely according to
ⱖ1 of the CDRs or an abnormal D-dimer result), CT was
indicated to confirm or exclude the diagnosis. Patients with
CT indicating PE received anticoagulants; this treatment was
withheld from all patients in whom the diagnosis was ex-
cluded. These latter patients were followed for 3 months. Fig-
ure 1 shows the study flow.
Standard contrast-enhanced multidetector CT was
done by using a 4-slice, 16-slice, or 64-slice multidetector
CT scanner with acquisition of 0.5- or 1-mm sections (de-
pending on the weight of the patient) of the entire chest
for diagnosing or excluding PE. The rotation time was 0.4
second, and the pitch factor was 1.4. The tube current was
250 to 300 mA, and the tube voltage was 100 kV. Acqui-
sitions were done during a single breath-hold lasting 10 to
12 seconds or less, depending on the type of scanner.
Eighty to 100 mL of contrast agent was injected in the
antecubital vein at an injection rate of 4.0 mL/sec. Acqui-
sition of the static pulmonary angiography scan was started
after detection of automated threshold enhancement in
the pulmonary trunk. A threshold difference of 100
Hounsfield units was selected for starting the acquisition.
Skilled radiologists read the CT scans to determine
whether PE was present or could be excluded. The radiol-
ogists were aware of an indication for CT but not whether
this was based on a high CDR result, an elevated D-dimer
Figure 1. Study flow diagram.
The items of 4 CDRs were collected:
Wells rule
Simplified Wells rule
Revised Geneva score
Simplified revised Geneva score
Computer program
The probability of PE was calculated on the
basis of each CDR: PE unlikely or PE likely
This was combined with the D-dimer test result
A highly sensitive quantitative D-dimer test was
done in all patients
No treatment for VTE;
3-mo follow-up
PE excluded
Normal
D-Dimer test
All rules: PE unlikely ≥1 rule was PE likely (discordant
outcomes among the CDRs) All rules: PE likely
No treatment for VTE;
3-mo follow-up
PE excluded
Abnormal
Further testing
Inconclusive
CT done
Treatment of PE
PE diagnosed
CDR ⫽clinical decision rule; CT ⫽computed tomography; PE ⫽pulmonary embolism; VTE ⫽venous thromboembolism.
Original ResearchClinical Decision Rules in the Management of Acute Pulmonary Embolism
www.annals.org 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 711
result, or both. The diagnosis of PE was confirmed by the
presence of at least 1 filling defect in the pulmonary artery
tree. Management of patients with an inconclusive CT
scan result was left to the attending physician and could
include repeated CT, V
˙/Q
˙scintigraphy, or conventional
pulmonary angiography.
Computerized Program
The treating physicians did clinical evaluations and
collected data at baseline. In each participating center, a
study coordinator was available for advice about the study.
This coordinator also checked the completeness and cor-
rectness of the data. Demographic data and additional rel-
evant information (for example, recent trauma or surgery,
cancer, use of anticoagulant drugs, duration of time since
symptom onset, and D-dimer result) were collected on a
case-report form that was available in paper and digital
format. The computerized design forced the physician to
start the diagnostic process with a clinical evaluation of the
patient and to enter all variables necessary to calculate the 4
CDRs and the D-dimer result into the computer. The
computer program calculated the 4 individual CDR scores
and, after combining these scores with the D-dimer test
result, indicated the next recommended step in the diag-
nostic process according to the predefined study flow: ex-
clusion of PE on the basis of CDR and D-dimer result, or
need to perform CT (Figure 1).
Follow-up
Patients in whom PE was excluded were followed for 3
months, either on the basis of the CDR and D-dimer com-
bination (for all 4 CDRs) or on the basis of a normal CT
scan. All patients were instructed to return to the hospital
if symptoms of venous thromboembolism (VTE) (PE or
deep venous thrombosis [DVT]) or bleeding events oc-
curred. Objective diagnostic tests (for example, CT, V
˙/Q
˙
scintigraphy, or compression ultrasonography) were done
if VTE was suspected. One of the study coordinators in-
terviewed patients by telephone at the end of 3 months and
questioned patients about health-related events during the
previous 3 months, especially for symptoms suggestive of
PE or DVT, interval initiation of anticoagulant drugs, and
possible hemorrhagic complications. If relevant, the pa-
tient’s general practitioner was contacted for additional
information. If a patient died, the cause of death was
obtained from hospital records, autopsy reports, or infor-
mation from the general practitioner. Deaths were classi-
fied as caused by PE if the cause of death was confirmed by
an autopsy report, if an objective diagnostic test was posi-
tive for PE before death, or if the cause of death could not
completely be explained by reasons other than VTE. A
panel of 3 experts adjudicated all outcomes.
Statistical Analysis
On the basis of a

value of 10% (power, 90%) and an
␣
value of 0.05, we calculated that 128 positive CT results
would be needed to detect a difference in sensitivity of
more than 5 percentage points (55% vs. 50%) between the
2 primary CDRs (the Wells rule and the revised Geneva
score). On the basis of a 20% prevalence of PE (2), a
sample size of 753 participants with suspected PE was re-
quired. All additional sample-size calculations on other
outcomes necessitated a smaller sample size. Because of
possible withdrawal, we aimed for a total sample size of
800 patients.
In this study, the 4 CDRs were directly compared for
their performance in determining whether patients had PE.
This included 4 primary analyses: 1) the ability of each CDR
to correctly categorize patients with suspected PE as unlikely
or likely; 2) the proportion of patients in whom PE was ex-
cluded on the basis of an unlikely CDR result combined with
a normal D-dimer test result at the time of the acute evalua-
tion; 3) the safety of clinical management based on each CDR
and D-dimer combination to exclude the diagnosis—true-
negative results (proportion of patients safely managed with-
out CT) and false-negative results (the VTE rate during the
3-month follow-up in patients in whom PE was considered
ruled out by the initial diagnostic work-up and who did not
receive anticoagulant drugs during follow-up); and 4) the dis-
tribution of patients in the probability categories according to
the 4 CDRs, which was studied by using sensitivity, specific-
ity, and receiving-operating characteristic analysis. Also, the
discordant cases (patients classified as unlikely by one CDR
but likely by another) were described. The reference standard
in patients in whom CT was not indicated was the recurrent
VTE rate during 3-month follow-up. For patients who
needed CT, the reference standard was CT and 3-month
follow-up.
Performance of the 4 CDRs and the combination of the
CDRs and D-dimer testing was examined by using sensitivity,
specificity, receiving-operating characteristic analysis, event
rates, and predictive values. To assess differences among the 4
CDRs in sensitivity, specificity, and predictive values and to
compare the categorization of patients into the probability
groups (paired data), the McNemar test was used to compare
each CDR with the other CDRs individually.
Furthermore, stratification by type of hospital (aca-
demic and nonacademic) was done to give insight into the
possible type of hospital-associated differences by using a
stratified Mantel–Haenszel test (CDR likely or unlikely vs.
outcome of PE stratified by academic versus nonacademic
hospitals). Exact 95% CIs around the observed incidences
were calculated by using CI analysis (13). Descriptive vari-
ables were calculated by using SPSS software, version 16.0
(SPSS, Chicago, Illinois). Mean values and frequencies,
such as the clinical characteristics of subgroups, were com-
pared by using a ttest and chi-square test, respectively.
Statistical significance was set at P⬍0.05.
Role of the Funding Source
This study was supported by unrestricted grants from
the Academic Medical Center, VU University Medical
Center, Rijnstate Hospital, Leiden University Medical
Center, Maastricht University Medical Center, Erasmus
Original Research Clinical Decision Rules in the Management of Acute Pulmonary Embolism
712 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 www.annals.org
Medical Center, and Maasstad Hospital. The boards of the
respective hospitals had no specific role in the design and
conduct of the study; collection, management, analysis,
and interpretation of the data; preparation, review, or ap-
proval of the manuscript; or the decision to submit the
manuscript for review.
RESULTS
Patients
From July 2008 to November 2009, a total of 1023
consecutive patients with clinically suspected PE were
screened, of whom 195 (19%) were excluded because of 1
or more of the predefined exclusion criteria (Figure 2). In
addition, 21 patients declined to give informed consent.
The final study population of 807 participants included
644 (80%) outpatients and 163 (20%) inpatients. The
baseline demographic and clinical characteristics of the 807
study participants are shown in Table 2.
Results of the Diagnostic Algorithm
Patients were managed according to the results of
CDRs combined with the D-dimer test result (Figure 2).
Discordant CDR results were observed in 243 patients
(29%), whereas results were concordant in 564 patients. In
total, PE was ruled out by a combination of an unlikely
CDR result according to all 4 CDRs and a normal D-dimer
result in 169 patients (21%). In 638 patients (79%), CT
was indicated either because of an abnormal D-dimer result
(265 patients) or because at least 1 of the CDRs indicated
that PE was likely (373 patients).
D-Dimer testing was not done in 19 patients (protocol
violations). This happened in 1 patient classified as PE-
unlikely according to all 4 CDRs. This patient was re-
garded as having a positive D-dimer result, and CTPA was
done (this patient was 1 of the 265 patients with an ab-
normal D-dimer result). In 18 other patients, the CDR
results were discordant. The missing D-dimer result had no
effect on the next step in the strategy because CTPA was
needed on the basis of discordant CDRs.
Protocol violations for CTPA occurred in 16 patients.
In 9 of these patients, CT was indicated but not done;
these patients were all followed for 3 months. In 7 patients,
CT was done even though it was not indicated, and it
showed PE in 1 of these patients.
In total, CT confirmed the diagnosis of PE in 185
patients: 184 with at least 1 CDR and positive D-dimer
result indicating PE-likely and 1 for whom CT was not
indicated on the basis of study criteria but was done be-
cause of clinical judgment (patient 1 in Appendix Table 1,
available at www.annals.org; Figure 2). The diagnosis was
excluded in 435 patients. In 164 patients, an alternative
diagnosis for the symptoms was found. Computed tomog-
raphy was inconclusive in 10 patients: Repeated CT ex-
cluded PE in 2 of these patients, and V
˙/Q
˙scintigraphy
excluded it in 2 other patients. In 1 patient, anticoagulant
treatment was started on the basis of an inconclusive CT
scan combined with high clinical suspicion of PE, and in
another, thrombosis of the subclavian vein was found on
the same scan; these patients received treatment accord-
ingly. In the remaining 4 patients with inconclusive CT
scans, the diagnosis was considered to be excluded without
further testing, and as a result, these patients did not re-
ceive treatment with anticoagulant medication. A final di-
agnosis was established within 1 hour in most of the pa-
tients or, at maximum, within 24 hours after presentation.
The overall prevalence of PE in the total study population
was therefore 185 of 807 patients (23% [CI, 20% to
26%]).
Follow-up
In 7 of 169 patients in the all-unlikely group who had
a normal D-dimer result, the protocol was violated and CT
was done even though it was not indicated. In 1 of these
patients, PE was diagnosed. This was regarded as a diag-
nostic failure in the CDR and D-dimer combination strat-
egy (1 of 169 patients; 0.6% [CI, 0.02% to 3.3%]) (pa-
tient 1 in Appendix Table 1;Figure 2). None of the
remaining 168 patients in this group received treatment
with anticoagulant drugs during follow-up, and all of these
patients had an uneventful follow-up. Nine patients in
whom CT was indicated but not done did not receive
treatment and had an uneventful follow-up. Of the 435
patients in whom PE was excluded with CT and the 8
patients who had inconclusive results and did not receive
treatment, 10 (2.3%) received anticoagulant drugs during
follow-up for reasons other than VTE. Seven of the 433
patients with a normal or inconclusive CT result who did
not receive anticoagulants for other reasons returned with
symptomatic and objectively confirmed VTE events during
3-month follow-up (patients 2 to 8 in Appendix Table 1;
Figure 2). Eighteen patients died during follow-up. In 1 of
these patients, DVT had already been diagnosed during
follow-up; in another patient, PE was excluded as the cause
of death by autopsy report; whereas in the remaining 16
patients, the cause of death was adjudicated to be unrelated
to a possible VTE. Therefore, the failure rate of a normal
or inconclusive CT scan in this study was 7 in 433 (1.6%
[CI, 0.7% to 3.3%]). One patient (of 807 [0.1%]) was lost
to follow-up. In a worst-case scenario in which this patient
would have developed VTE, the failure rate after CT that
excluded PE would have been 8 in 433 (1.9% [CI, 0.8% to
3.6%]). Allergy to the intravenous iodinated contrast or
contrast-induced nephropathy was not recognized in the
included patients during the study.
Categorization of Patients in Probability Groups With
the 4 CDRs
Table 3 shows how the patients were categorized by
the 2 probability categories of the 4 CDRs without taking
the D-dimer results into account. The proportion of pa-
tients classified as PE-unlikely was similar for the 4 CDRs.
Also, the prevalence of PE in the unlikely categories was
similar. Overall, the proportion of patients classified as PE-
Original ResearchClinical Decision Rules in the Management of Acute Pulmonary Embolism
www.annals.org 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 713
Figure 2. Results of the diagnostic strategy.
Excluded
No informed consent (n = 21)
Met ≥1 exclusion criteria (n = 195)*
Previous PE: 82
Treatment with vitamin K
antagonists: 41
>24 h use of LMWH: 20
Follow-up not possible: 19
Pregnancy: 18
Renal insufficiency: 14
Life expectancy <3 mo: 12
Allergy to intravenous contrast
agent: 4
Did not receive treatment
(n = 8)
Normal repeated CT: 2
Negative V
•
/Q
•
result: 2
No further testing: 4
Received treatment (n = 2)
Upper-extremity
thrombosis: 1
Physician’s discretion
or high clinical
suspicion: 1
Computer program
The outcome of 4 clinical decision rules and the
D
-dimer test result determined the next step
(Figure 1)
Study patients (n = 807)
Patients with
clinically suspected PE
(n = 1023)
Did not receive
treatment (n = 169)
PE excluded (n = 169)†
Normal (n = 169)
D
-Dimer test
All rules: PE unlikely
(n = 434)
Disagreement of the 4 rules
(≥1 rule PE likely) (n = 243)
All rules: PE likely
(n = 130)
Did not receive
treatment (n = 425)‡
Did not receive
treatment (n = 9)
Received treatment
(n = 184)
PE excluded (n = 435)
Abnormal (n = 265)
3-mo follow-up
VTE: 1
3-mo follow-up
VTE: 7
PE: 1
DVT: 6
Lost to follow-up: 1
3-mo follow-up
VTE: 0
3-mo follow-up
VTE: 0
Inconclusive (n = 10)CT not done (n = 9)
CT done (n = 638)
PE (n = 184)
PE: 1
CT ⫽computed tomography; DVT ⫽deep venous thrombosis; LMWH ⫽low-molecular-weight heparin; PE ⫽pulmonary embolism; V
˙/Q
˙⫽
ventilation–perfusion; VTE ⫽venous thromboembolism.
*Some of the 195 patients met more than 1 exclusion criterion.
†In 7 patients, CT was done although it was not indicated; CT confirmed the diagnosis of PE in 1 patient. “Received treatment” or “did not receive
treatment” refers to treatment with anticoagulant drugs.
‡Ten patients in whom PE was excluded by CT received anticoagulant treatment for reasons other than VTE.
Original Research Clinical Decision Rules in the Management of Acute Pulmonary Embolism
714 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 www.annals.org
likely was largest when the simplified Wells rule was used
38% versus 28% to 32% with the other 3 CDRs (Table
3). The sensitivity and specificity of each CDR alone
(without D-dimer results) ranged from 49% to 65% and
from 70% to 80%, respectively (Appendix Table 2, avail-
able at www.annals.org). The receiver-operating character-
istic curves for the 4 CDRs were similar, and areas under
the curve ranged from 0.69 to 0.73 (Appendix Figure 1,
available at www.annals.org).
Performance of the 4 CDRs With D-Dimer Results
Combined with a normal D-dimer result, the 4 CDRs
excluded PE in similar proportions of patients, ranging
from 22% to 24% (Table 3). The 3-month VTE failure
rates did not differ among the CDR and D-dimer test result
combinations and ranged from 0.5% to 0.6% (Table 3).
The 95% CI was 0% to 3% for all CDRs (Table 3). When
combined with the D-dimer test result, the sensitivities of
the various CDRs did not differ, although there were small
differences in specificity (Table 4).
Discordance Among the CDRs
Of the 434 patients with all 4 CDRs indicating PE-
unlikely, 52 (12%) received a diagnosis of PE; all patients
except 1 had an abnormal D-dimer result, the latter of
which indicated the need for CT.
In 243 of 807 patients (29%), discordance among CDRs
was observed (Figure 2). The D-dimer result was normal in 29
patients and abnormal in 196 patients, and the test was not
done in 18 patients. In the latter 243 patients, CT was done,
which confirmed PE in 1 patient.
The number of discordant cases between 2 scores ranged
from 25 of 807 patients (3.1%) between the revised Geneva
score and the simplified revised Geneva score to 199 of 807
patients (25%) between the Wells rule and the revised Geneva
score (Appendix Table 3, available at www.annals.org). The
agreement was greatest between the original scores (Wells rule
and revised Geneva score) and their simplified versions: Dis-
cordance was seen in 11% of the total cohort between the
Wells rule and the simplified Wells rule and 3.1% between
the revised Geneva score and simplified Geneva score. Discor-
dance was greater than 20% between all other scores.
Despite the discordant scores, PE was not missed in
any patients in the discordant group who had a normal
Table 2. Clinical Characteristics of Patients With Suspected
Pulmonary Embolism
Characteristic Value
Mean age (SD), y53 (17.7)
Women, n (%) 487 (60.3)
Outpatient, n (%) 644 (79.8)
Median duration of symptoms (IQR), d2 (1–7)
Mean body mass index (SD), kg/m
2
26.3 (5.5)
Risk factors,
n (%)
Immobilization or recent surgery 176 (21.8)
Previous venous thromboembolism 39 (4.8)
Chronic obstructive pulmonary disease with treatment 75 (9.3)
Heart failure with treatment 47 (5.8)
Active cancer 114 (14.1)
Estrogen use by women 97 (19.9)
Body mass index ⱖ30 kg/m
2
152 (4.6)
Symptoms and clinical presentation
Clinical symptoms of deep venous thrombosis, n (%) 47 (5.8)
Mean heart rate (SD), beats/min 88 (18.8)
Hemoptysis, n (%) 40 (5.0)
IQR ⫽interquartile range.
Table 3. Patients With Unlikely or Likely Clinical Probability of PE on the Basis of 4 CDRs and a CDR Plus D-Dimer Test
(
n
ⴝ807)
Variable Original
Wells Rule
Simplified
Wells Rule
Original Revised
Geneva Score
Simplified Revised
Geneva Score
CDR unlikely
Number 584 499 553 576
Percentage (95% CI) 72 (69–76) 62 (59–65) 69 (65–72) 71 (68–75)
Prevalence of PE in CDR-unlikely patients
Number/number 90/584 65/499 88/553 95/576
Percentage (95% CI) 15 (13–18) 13 (10–16) 16 (13–19) 17 (14–20)
CDR likely
Number 223 308 254 231
Percentage (95% CI) 28 (25–31) 38 (35–41) 32 (28–35) 29 (26–32)
Prevalence of PE in CDR-likely patients
Number/number 95/223 120/308 97/254 90/231
Percentage (95% CI) 43 (36–49) 39 (34–44) 38 (32–44) 39 (32–45)
CDR unlikely and normal D-dimer result
Number 184 178 185 190
Percentage (95% CI) 23 (20–26) 22 (19–25) 23 (20–26) 24 (21–27)
Incidence of venous thromboembolism in CDR-unlikely
patients with a normal D-dimer result
Number/number 1/184 1/178 1/185 1/190
Percentage (95% CI) 0.5 (0.0–3.0) 0.6 (0.0–3.1) 0.5 (0.0–3.0) 0.5 (0.0–2.9)
CDR ⫽clinical decision rule; PE ⫽pulmonary embolism.
Original ResearchClinical Decision Rules in the Management of Acute Pulmonary Embolism
www.annals.org 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 715
D-dimer result (Table 5). Therefore, when combined with
D-dimer results, the scores performed equally well in ex-
cluding PE.
Inpatients
Both inpatients and outpatients were included. The pro-
portions of inpatients who were categorized as PE-unlikely
were 37% for the simplified Wells rule, 48% for the revised
Geneva score, 50% for the simplified revised Geneva score,
and 57% for the Wells rule. These proportions were smaller
than the proportions of outpatients categorized as PE-unlike-
ly: 68%, 74%, 77%, and 76% for the 4 CDRs, respectively
(multiple tests; all P⬍0.01).
The failure rate of excluding PE on the basis of an
unlikely CDR and a normal D-dimer result was similar
for both inpatients and outpatients with all 4 CDRs.
However, the proportion of inpatients in which PE
could be excluded noninvasively was very low: Only 3
inpatients for the simplified Wells rule (3 of 163 pa-
tients [1.8%]), 4 patients for the Wells rule (2.5%), and
5 patients for the revised Geneva score and the simpli-
fied revised Geneva score (3.1%). No failures occurred
in the inpatients in whom PE was excluded without the
need for CTPA.
Stratification by Academic Versus Nonacademic
Hospitals
In total, 5 academic hospitals included 598 (74%) pa-
tients, whereas 2 nonacademic hospitals included 209 patients
(26%). Demographic characteristics (Table 2) did not differ
for patients from academic versus nonacademic hospitals, ex-
cept for cancer (16% vs. 8.1%; P⬍0.001) and recent surgery
or immobilization (26% vs. 11%; P⬍0.001). After the re-
sults for academic and nonacademic hospitals were adjusted
for, probability (PE-unlikely or PE-likely) was correctly cate-
gorized more often at nonacademic sites (75% to 79% versus
66% to 71% at academic hospitals; P⬍0.001 for all 4
CDRs).
Table 4. Accuracy Indexes of the Clinical Decision Rules in Combination With a Normal D-Dimer Result in Patients With a
Suspected Event*
Variable Original Wells Rule
(
n
ⴝ796)
Simplified Wells Rule
(
n
ⴝ803)
RGS (
n
ⴝ796) Simplified
RGS (
n
ⴝ795)
Sensitivity†
Number/number 190/191 191/192 188/189 187/188
Percentage (95% CI) 99.5 (97–100) 99.5 (97–100) 99.5 (97–100) 99.5 (97–100)
Specificity‡
Number/number 183/605 177/611 184/607 189/607
Percentage (95% CI) 30 (27–34) 29 (25–33) 30 (27–34) 31 (28–34)
Negative predictive value§
Number/number 183/184 177/178 184/185 189/190
Percentage (95% CI) 99.5 (97–100) 99.4 (97–100) 99.5 (97–100) 99.5 (97–100)
RGS ⫽revised Geneva rule.
*Patients with a clinical decision rule indicating that PE was unlikely but in whom the D-dimer result was missing (protocol violation) were not included in this analysis;
this number differed among the 4 clinical decision rules. Sensitivities did not differ among the 4 clinical decision rules in combination with D-dimer test. Specificity differed
significantly between the Wells rule and the simplified Wells rule (P⫽0.031) and the simplified Wells rule and the simplified RGS (P⫽0.017). Other differences in
specificity were not statistically significant.
†The number of patients correctly identified as having pulmonary embolism by the combination of clinical decision rules and D-dimer testing divided by the total number
of patients with proven pulmonary embolism identified by computed tomography at the time of initial evaluation or venous thromboembolism at 3-mo follow-up.
‡ The number of patients correctly identified as not having pulmonary embolism by the combination of clinical decision rules and D-dimer testing divided by the total
number of patients in whom pulmonary embolism was excluded by computed tomography at the time of initial evaluation or venous thromboembolism at 3-mo follow-up.
§The number of patients correctly identified as not having pulmonary embolism by the combination of clinical decision rules and D-dimer testing divided by the total
number of patients with the combination of clinical decision rule and D-dimer testing indicating that pulmonary embolism was excluded (i.e., pulmonary embolism and deep
venous thrombosis).
Table 5. Discordances Between the Categorization of Unlikely and Likely Clinical Probability Groups According to 4 Clinical
Decision Rules in 205 Patients With Suspected PE and a Normal D-Dimer Result*
Variable Wells Likely (
n
ⴝ21) SW Likely (
n
ⴝ27) RGS Likely (
n
ⴝ20) SRGS Likely (
n
ⴝ15)
Patients,
n
Patients With
PE,
n
Patients,
n
Patients With
PE,
n
Patients,
n
Patients With
PE,
n
Patients,
n
Patients With
PE,
n
Wells unlikely (n⫽184) 6 0 12 0 8 0
SW unlikely (n⫽178) 0 0 9 0 5 0
RGS unlikely (n⫽185) 13 0 16 0 0 0
SRGS unlikely (n⫽190) 14 0 17 0 5 0
PE ⫽pulmonary embolism; RGS ⫽revised Geneva score; SRGS ⫽simplified revised Geneva score; SW ⫽simplified Wells rule; Wells ⫽original Wells rule.
*In total, 29 patients had discordant clinical decision rule results.
Original Research Clinical Decision Rules in the Management of Acute Pulmonary Embolism
716 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 www.annals.org
DISCUSSION
Our accuracy study directly compared 4 CDRs to as-
sess the probability of PE and showed that the CDRs are
similar in 1) their ability to categorize patients as having
unlikely or likely clinical probability, 2) the proportion of
patients in whom CTPA was not indicated on the basis of
an unlikely CDR result and a normal D-dimer result, and
3) the 3-month failure rate for VTE in patients in whom
PE was excluded by CDR and D-dimer testing. Of note,
although discordance in the categorization of patients as
having unlikely or likely probability by the scores was pres-
ent in 30% of the patients, this did not result in a differ-
ence in failure rates when the CDR results were combined
with the D-dimer results.
Our results are important for and relevant to clinical
practice. Despite the debate on the subjective variable in
the Wells rule, in this direct comparison the Wells rule and
the simplified Wells rule were equivalent in performance
compared with the fully objective revised Geneva score. In
addition, we were able to validate the performance of the
recently introduced simplifications of the Wells rule and
revised Geneva score. Both simplified scores had similar
diagnostic performance compared with their original and
extensively validated versions. Despite discordances be-
tween the CDR outcomes in 30% of patients, there was no
difference in safety when using a management strategy
based on any of the CDRs combined with D-dimer testing.
This equal performance could be explained by the use of a
highly sensitive D-dimer test in patients with a CDR indi-
cation of PE-unlikely.
The importance of estimation of clinical probability has
been emphasized (14–18). Although D-dimer assay is sensitive
in the diagnosis of PE, false-negative results are more likely
when the pretest clinical probability is high (18). This pro-
spective validation of the simplified CDRs has relevant and
practical implications because they enable easier computation
of the clinical probability score, which in turn could lead to
better implementation of CDRs in daily clinical care.
Our findings are in line with previous studies that used
these CDRs in a 2-category scheme (14). With the Wells
rule, 51% to 84% of patients were categorized as PE-
unlikely in previous reports, with a prevalence of PE rang-
ing from 3.4% to 12%, compared with 72% categorized as
PE-unlikely in this study and a 15% prevalence of PE. By
using the simplified Wells rule, the proportion of patients
classified as PE-unlikely was slightly lower in our cohort
(62%) than in a previous validation study (78%), but the
prevalence of PE in this PE-unlikely group was similar
(13% in both studies) (10). Likewise, in an earlier retrospec-
tive study, the simplified revised Geneva score classified 65%
of patients as PE-unlikely compared with 62% in the current
analysis, with a 13% prevalence of PE in the previous study
and a 16% prevalence of PE in our study (8). Because we
believe that this is the first study to report a 2-category scheme
for the revised Geneva score, we cannot compare our data on
the 2-category revised Geneva score with previous findings.
However, the 69% of patients with an unlikely CDR result
according to the revised Geneva score and the 16% prevalence
of PE in this group overlap well with the results from the
other 3 decision rules.
Four highly sensitive but different D-dimer assays were
used. Because CDRs were in all patients, the types of
D-dimer assays were equally represented in the different
CDR groups, which enabled comparison of the CDRs re-
gardless of the D-dimer assay. The type of assay was not
based on randomization but depended on the preference of
the study center. Therefore, comparisons among the
D-dimer assays are limited by the sample sizes.
After several retrospective or small prospective com-
parisons, to our knowledge this is the first large study to
directly compare the most widely used CDRs (original
Wells rule and original revised Geneva score) in the diag-
nostic management of PE. Furthermore, our study pro-
spectively validated the performance of the recently intro-
duced simplified Wells rule and simplified revised Geneva
score. Calculation of the scores in all patients allowed di-
rect comparison of the CDRs in a single patient popula-
tion. Also, because of the computer-aided design of the
study, calculation errors were minimized. Likewise, the use
of a computer program to guide the physician to the next
step in the diagnostic algorithm excluded the possibility
that the physician’s preference for a certain CDR would
influence the management of a patient.
The results of our study may be applicable in a wide
range of clinical settings. First, the clinical characteristics of
the patients are similar to those in other population-based
studies (2, 4), and the 23% prevalence of PE in this cohort is
similar to that in other reports (2, 4, 6). In addition, consec-
utive patients were included from both academic and nonac-
ademic medical centers.
Our study has potential limitations. First, a random-
ized, controlled trial of the 4 CDRs is an alternative study
design, but in view of the reasonably high concordance
rates, it would probably have been very inefficient. In ad-
dition, by study design, CT was done in all patients with
discordant CDRs and ensured that an imaging diagnosis
was available in all of those patients. The diagnostic pro-
tocol was violated in 4 patients, in whom CT was not done
despite discordance of the CDRs. Three-month follow-up,
however, was uneventful in these patients. Second, manag-
ment was not based on 1 CDR in combination with
D-dimer testing; rather, it was based on the combination of
the 4 CDRs and D-dimer testing. According to the proto-
col, all patients with discordant CDR results had CT.
Most of these patients had elevated D-dimer levels and
would have an indication for CT, even if only 1 of the
CDRs was used for decision making. Only patients with
discordant results and a normal D-dimer level (29 patients
[3.6% of the included patients]) did not have an indication
for CT according to 1 of the separate rules combined with
D-dimer testing. These patients had CT because at least 1
Original ResearchClinical Decision Rules in the Management of Acute Pulmonary Embolism
www.annals.org 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 717
of the other rules indicated that PE was likely. Third, use
of a computerized decision-support system improves the
diagnosis of PE (19). However, in daily clinical practice,
these systems may not be widely available, and our results
may therefore differ from a setting in which more miscal-
culations are possible. Finally, although both inpatients
and outpatients were included in this study and no failures
occurred in the patients in whom PE could be excluded,
we could not validate that any of the CDR and D-dimer
combinations can safely exclude the diagnosis in inpatients.
Further research may include an outcome study using
1 of the simplified CDRs in combination with D-dimer
testing. Because patients with suspected recurrent PE were
not included, the performance of the CDRs in this group
will need additional research.
In conclusion, the Wells rule, the revised Geneva
score, the simplified Wells rule, and the simplified revised
Geneva score, in combination with a D-dimer test result, all
performed similarly in the exclusion of acute PE. This pro-
spective validation indicates that the simplified, more
straightforward CDRs may be used in clinical practice.
Which rule a physician will use should depend on local
preference and acquaintance to accomplish correct use of
the CDR and prevent miscalculations.
From Academic Medical Center and VU University Medical Center,
Amsterdam; Leiden University Medical Center, Leiden; School for Pub-
lic Health and Primary Care, Maastricht University Medical Center and
Cardiovascular Research Institute Maastricht, Maastricht; Rijnstate Hos-
pital Arnhem, Arnhem; and Erasmus University Medical Center and
Maasstad Hospital, Rotterdam, the Netherlands.
Note: Drs. Douma and Mos contributed equally to this work.
Grant Support: Dr. Mos is supported by a grant from the Netherlands
Heart Foundation (2006B224).
Potential Conflicts of Interest: Dr. Mos: Grant: The Netherlands
Heart Foundation. Dr. ten Cate: Consultancy: Bayer, Roche, Philips.
Payment for lectures including service on speakers bureaus: Bayer, Boehr-
inger Ingelheim, GlaxoSmithKline, Philips. Disclosures can also be
viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms
.do?msNum⫽M10-1964.
Reproducible Research Statement: Study protocol, statistical code, and
data set: Available from Dr. Mos (e-mail, i.c.m.mos@lumc.nl).
Requests for Single Reprints: Inge C.M. Mos, MD, Section of Vascu-
lar Medicine, Department of General Internal Medicine-Endocrinology,
Room C4-70, Leiden University Medical Center, PO Box 9600, 2300
RC Leiden, the Netherlands; e-mail, i.c.m.mos@lumc.nl.
Current author addresses and author contributions are available at www
.annals.org.
References
1. Wells PS, Anderson DR, Rodger M, Stiell I, Dreyer JF, Barnes D, et al.
Excluding pulmonary embolism at the bedside without diagnostic imaging: man-
agement of patients with suspected pulmonary embolism presenting to the emer-
gency department by using a simple clinical model and D-dimer. Ann Intern
Med. 2001;135:98-107. [PMID: 11453709]
2. van Belle A, Bu¨ ller HR, Huisman MV, Huisman PM, Kaasjager K, Kam-
phuisen PW, et al; Christopher Study Investigators. Effectiveness of managing
suspected pulmonary embolism using an algorithm combining clinical probabil-
ity, D-dimer testing, and computed tomography. JAMA. 2006;295:172-9.
[PMID: 16403929]
3. Perrier A, Roy PM, Sanchez O, Le Gal G, Meyer G, Gourdier AL, et al.
Multidetector-row computed tomography in suspected pulmonary embolism. N
Engl J Med. 2005;352:1760-8. [PMID: 15858185]
4. Ten Cate-Hoek AJ, Prins MH. Management studies using a combination of
D-dimer test result and clinical probability to rule out venous thromboembolism:
a systematic review. J Thromb Haemost. 2005;3:2465-70. [PMID: 16150049]
5. Wells PS, Anderson DR, Rodger M, Ginsberg JS, Kearon C, Gent M, et al.
Derivation of a simple clinical model to categorize patients probability of pulmo-
nary embolism: increasing the models utility with the SimpliRED D-dimer.
Thromb Haemost. 2000;83:416-20. [PMID: 10744147]
6. Le Gal G, Righini M, Roy PM, Sanchez O, Aujesky D, Bounameaux H, et
al. Prediction of pulmonary embolism in the emergency department: the revised
Geneva score. Ann Intern Med. 2006;144:165-71. [PMID: 16461960]
7. Gibson NS, Sohne M, Kruip MJ, Tick LW, Gerdes VE, Bossuyt PM, et al;
Christopher Study Investigators. Further validation and simplification of the
Wells clinical decision rule in pulmonary embolism. Thromb Haemost. 2008;99:
229-34. [PMID: 18217159]
8. Klok FA, Mos IC, Nijkeuter M, Righini M, Perrier A, Le Gal G, et al.
Simplification of the revised Geneva score for assessing clinical probability of
pulmonary embolism. Arch Intern Med. 2008;168:2131-6. [PMID: 18955643]
9. Klok FA, Kruisman E, Spaan J, Nijkeuter M, Righini M, Aujesky D, et al.
Comparison of the revised Geneva score with the Wells rule for assessing clinical
probability of pulmonary embolism. J Thromb Haemost. 2008;6:40-4. [PMID:
17973649]
10. Douma RA, Gibson NS, Gerdes VE, Bu¨ ller HR, Wells PS, Perrier A, et al.
Validity and clinical utility of the simplified Wells rule for assessing clinical prob-
ability for the exclusion of pulmonary embolism. Thromb Haemost. 2009;101:
197-200. [PMID: 19132208]
11. Calisir C, Yavas US, Ozkan IR, Alatas F, Cevik A, Ergun N, et al. Perfor-
mance of the Wells and Revised Geneva scores for predicting pulmonary embo-
lism. Eur J Emerg Med. 2009;16:49-52. [PMID: 18931619]
12. Wong DD, Ramaseshan G, Mendelson RM. Comparison of the Wells and
revised Geneva scores for the diagnosis of pulmonary embolism: an Australian
experience. Intern Med J. 2011;41:258-63. [PMID: 20214691]
13. Gardner, MJ, Altman, DG, eds. Statistics with Confidence. Confidence
Intervals and Statistical Guidelines. London: BMJ; 1989:59-60.
14. Ceriani E, Combescure C, Le Gal G, Nendaz M, Perneger T, Bou-
nameaux H, et al. Clinical prediction rules for pulmonary embolism: a
systematic review and meta-analysis. J Thromb Haemost. 2010;8:957-70.
[PMID: 20149072]
15. Roy PM, Meyer G, Vielle B, Le Gall C, Verschuren F, Carpentier F, et al;
EMDEPU Study Group. Appropriateness of diagnostic management and out-
comes of suspected pulmonary embolism. Ann Intern Med. 2006;144:157-64.
[PMID: 16461959]
16. Kelly J, Hunt BJ. A clinical probability assessment and D-dimer measurement
should be the initial step in the investigation of suspected venous thromboembo-
lism. Chest. 2003;124:1116-9. [PMID: 12970044]
17. Righini M, Aujesky D, Roy PM, Cornuz J, de Moerloose P, Bounameaux
H, et al. Clinical usefulness of D-dimer depending on clinical probability and
cutoff value in outpatients with suspected pulmonary embolism. Arch Intern
Med. 2004;164:2483-7. [PMID: 15596640]
18. Gibson NS, Sohne M, Gerdes VE, Nijkeuter M, Buller HR. The impor-
tance of clinical probability assessment in interpreting a normal D-dimer in pa-
tients with suspected pulmonary embolism. Chest. 2008;134:789-93. [PMID:
18641091]
19. Roy PM, Durieux P, Gillaizeau F, Legall C, Armand-Perroux A, Martino
L, et al. A computerized handheld decision-support system to improve pulmo-
nary embolism diagnosis: a randomized trial. Ann Intern Med. 2009;151:677-86.
[PMID: 19920268]
Original Research Clinical Decision Rules in the Management of Acute Pulmonary Embolism
718 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 www.annals.org
Current Author Addresses: Drs. Douma, Bu¨ller, and Kamphuisen: De-
partment of Vascular Medicine, Academic Medical Center, Meibergdreef
9, 1105 AZ, the Netherlands.
Dr. Mos: Section of Vascular Medicine, Department of General Internal
Medicine-Endocrinology, Room C4-70, Leiden University Medical
Center, PO Box 9600, 2300 RC Leiden, the Netherlands.
Ms. Erkens: Cardiovascular Research Institute Maastricht, P. Debyeplein
1, 6200 AZ Maastricht, the Netherlands.
Dr. Nizet: Department of Pulmonary Diseases, MC Haagenlande West-
einde, PO Box 432, 2501 CK Den Haag, the Netherlands.
Dr. Durian: Department of Hematology, Erasmus MC, PO Box 2040,
3000 CA Rotterdam, the Netherlands.
Dr. Hovens: Department of Internal Medicine, Rijnstate Hospital, Wag-
nerlaan 55, 6815 AD Arnhem, the Netherlands.
Dr. van Houten: Department of Internal Medicine, Maasstad Hospital,
Olympiaweg 350, 3078 HT Rotterdam, the Netherlands.
Dr. Hofstee: Department of Internal Medicine, VU University Medical
Center, PO Box 7057, 1007 MB Amsterdam, the Netherlands.
Dr. Klok: Section of Vascular Medicine, Department of General Internal
Medicine and Endocrinology, Room C4-68, PO Box 9600, Leiden Uni-
versity Medical Center, Albinusdreef 2, 2300 RC Leiden, the Nether-
lands.
Dr. ten Cate: Laboratory for Clinical Thrombosis and Hemostasis, De-
partment of Internal Medicine and Cardiovascular Research Institute
Maastricht, Maastricht University Medical Center, PO Box 616, UNS
50/Box 8, 6200 MD Maastricht, the Netherlands.
Dr. Ullmann: Rijnstate Hospital, 6815 AD Arnhem, the Netherlands,
Dr. Huisman: Section of Vascular Medicine, Department of General
Internal Medicine-Endocrinology, Room C4-68, Leiden University
Medical Center, PO Box 9600, 2300 RC Leiden, the Netherlands.
Author Contributions: Conception and design: R.A. Douma, I.C.M.
Mos, T.A.C. Nizet, H.M.A. Hofstee, F.A. Klok, E.F. Ullmann, H.R.
Bu¨ller, P.W. Kamphuisen, M.V. Huisman.
Analysis and interpretation of the data: R.A. Douma, I.C.M. Mos, H.R.
Bu¨ller, P.W. Kamphuisen, M.V. Huisman.
Drafting of the article: R.A. Douma, I.C.M. Mos, T.A.C. Nizet, A.A.
van Houten, H. ten Cate, P.W. Kamphuisen.
Critical revision of the article for important intellectual content: R.A.
Douma, I.C.M. Mos, P.M.G. Erkens, T.A.C. Nizet, M.F. Durian,
M.M. Hovens, F.A. Klok, H. ten Cate, E.F. Ullmann, H.R. Bu¨ller, P.W.
Kamphuisen, M.V. Huisman.
Final approval of the article: R.A. Douma, I.C.M. Mos, T.A.C. Nizet,
M.F. Durian, M.M. Hovens, A.A. van Houten, F.A. Klok, H. ten Cate,
H.R. Bu¨ller, P.W. Kamphuisen, M.V. Huisman.
Provision of study materials or patients: I.C.M. Mos, P.M.G. Erkens,
T.A.C. Nizet, M.M. Hovens, H.M.A. Hofstee.
Statistical expertise: F.A. Klok, E.F
Administrative, technical, or logistic support: R.A. Douma, I.C.M. Mos,
T.A.C. Nizet, M.F. Durian, A.A. van Houten, E.F. Ullmann.
Collection and assembly of data: R.A. Douma, I.C.M. Mos, P.M.G.
Erkens, T.A.C. Nizet, M.F. Durian, M.M. Hovens, A.A. van Houten,
H.M.A. Hofstee, E.F. Ullmann, P.W. Kamphuisen.
APPENDIX 1: THE PROMETHEUS STUDY GROUP
Academic Medical Center: L.F. Beenen, P.M.M. Bossuyt,
H.R. Bu¨ller, E.J. van den Dool, R.A. Douma, and P.W. Kam-
phuisen; VU University Medical Center: H.M.A. Hofstee and
M.H.H. Kramer; Rijnstate Hospital: M.M. Hovens, K. Kaasjager,
T.A.C. Nizet, and E.F. Ullmann; Leiden University Medical Cen-
ter: H.C.J. Eikenboom, C. Heringhaus, M.V. Huisman, F.A.
Klok, K.W. van Kralingen, L.J.M. Kroft, I.C.M. Mos, and A. de
Roos; Maastricht University Medical Center: K. Boomars, H. ten
Cate, and P.M.G. Erkens; Erasmus Medical Center: M.F. Durian,
M.J.H.A. Kruip; Maasstad Hospital: A.A. van Houten.
APPENDIX 2: DICHOTOMIZATION OF THE REVISED
GENEVA SCORE
The revised Geneva score was dichotomized before the start
of this study. The other 3 CDRs already have a dichotomized
score, and the participating hospitals were already using 2-level
CDRs, so it was easier to incorporate a dichotomized score in this
study design. We searched for the optimal cutoff level by using
an existing cohort of patients with suspected PE (2) for whom
the revised Geneva score variables were available for calculation
of the score. We chose the optimal cutoff on the basis of the
optimal combination between sensitivity and specificity, receiver-
operating characteristic curve analysis, and frequency tables. As
can be seen in Appendix Table 4 and Appendix Figure 2,a
cutoff value of 5 or less revealed an area under the curve of 0.664.
With this cutoff, the prevalence of PE in the unlikely category
was 10% compared with 32% in the likely category, which is
similar to rates of PE in other dichotomous schemes (14).
Furthermore, from a practical standpoint, by using a cutoff of
5 or less, a substantial proportion (70%) of patients had a
CDR outcome below this cutoff, whereas only 30% had a
CDR outcome above the cutoff. Adhering to the diagnostic
algorithm, 30% of patients would have to proceed directly to
CTPA. In comparison, with a cutoff value of 4 or less, the
CDR indicated that PE was likely in 45% of patients, mean-
ing that they would all directly have an indication for CTPA.
The area under the receiver-operating characteristic curve did
not differ significantly with that of a cutoff of 5 or less (0.668
vs. 0.664) (Appendix Figure 2).
Annals of Internal Medicine
W-250 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 www.annals.org
Appendix Table 1. Characteristics of Patients in Whom VTE Was Found During the 3-Month Follow-up, Despite Initial Exclusion of
the Diagnosis
Patient Characteristic Outcome of Diagnostic Tests at Inclusion Follow-up Brief Description
Sex Age,
y
Wells SW RGS SRGS DD CT at
Presentation
VTE Day
1 Male 65 2222 482 Indicating PE PE 0 CT was done although it was not indicated (all
CDRs unlikely and a normal D-dimer result) and
was positive for PE; multiple subsegmental emboli
were found, as well as signs suggesting
pulmonary infarction
2 Female 63 11111535 Normal DVT 19 Also suspected DVT at presentation; CUS was
negative for DVT at presentation
3 Male 63 21111100 Normal PE 22 PE found on CT done for other reasons
4 Female 39 22221100 Normal DVT 27 DVT was found on CUS at 27 d (during
hospitalization); despite this finding,
anticoagulation was delayed until 51 d after
another positive CUS result for DVT
5 Female 58 22222600 Normal DVT 62 DVT of jugular and subclavian vein, patient had
Takayasu arteritis
6 Female 43 21222100 Normal DVT 21 DVT of jugular vein found by coincidence on
staging CT after chemoradiotherapy
7 Male 87 22113420 Normal DVT 0 DVT found on CUS done directly after a negative
CT scan for PE; patient also had leg symptoms
8 Female 62 1122 –* Normal DVT 7 DVT after surgery and immobilization
CDR ⫽clinical decision rule; CUS ⫽compression ultrasonography; CT ⫽computed tomography; DD ⫽D-dimer test; DVT ⫽deep venous thrombosis; PE ⫽pulmonary
embolism; RGS ⫽revised Geneva score; SRGS ⫽simplified revised Geneva score; SW ⫽simplified Wells rule; VTE ⫽venous thromboembolism; Wells ⫽original Wells
rule; 1⫽CDR indicating that PE was likely; 2⫽CDR indicating that PE was unlikely.
*D-Dimer testing was not done.
Appendix Table 2. Accuracy Indexes of the Clinical Decision Rules Alone in 807 Patients With a Suspected Event*
Variable Wells Rule Simplified Wells Rule RGS Simplified RGS
Sensitivity†
Number/number 99/192 125/192 101/192 94/192
Percentage (95% CI) 52 (45–59) 65 (58–72) 53 (46–60) 49 (42–56)
Specificity‡
Number 491/615 432/615 462/615 478/615
Percentage (95% CI) 80 (77–83) 70 (67–74) 75 (72–79) 78 (74–81)
Negative predictive value§
Number 491/584 432/499 462/553 478/576
Percentage (95% CI) 84 (81–87) 87 (84–90) 84 (81–87) 83 (80–86)
RGS ⫽revised Geneva score.
*Sensitivity differed significantly between the Wells rule and the simplified Wells rule (P⬍0.001), the simplified Wells rule and the RGS (P⫽0.001), the simplified Wells
rule and the simplified RGS (P⬍0.001), and the RGS and the simplified RGS (P⫽0.039). Other differences in sensitivity were not statistically significant.
†The number of patients correctly identified as having pulmonary embolism by the clinical decision rules alone (independent of D-dimer results) divided by the total number
of patients with proven pulmonary embolism identified by computed tomography at the time of initial evaluation or venous thromboembolism at 3-mo follow-up.
‡The number of patients correctly identified as not having pulmonary embolism by the clinical decision rules alone (independent of D-dimer results) divided by the total
number of patients in whom pulmonary embolism was excluded by computed tomography at the time of initial evaluation or venous thromboembolism at 3-mo follow-up.
§The number of patients correctly identified as not having pulmonary embolism on the basis of the clinical decision rules alone (independent of D-dimer results) divided
by the total number of patients with a clinical decision rule result of unlikely (i.e., pulmonary embolism and deep venous thrombosis).
www.annals.org 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 W-251
Appendix Figure 2. Receiver-operating characteristic curves at
different RGS cutoffs.
Sensitivity
1 – Specificity
RGS cutoff ≤4
RGS cutoff ≤5
RGS cutoff ≤6
RGS cutoff ≤7
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
The area under the receiving-operating characteristic curve was similar
when 4 different RGS cutoff levels were applied: 0.67 (95% CI, 0.59 to
0.75) for a cutoff ⱕ4, 0.66 (CI, 0.58 to 0.75) for a cutoff ⱕ5, 0.65 (CI,
0.56 to 0.74) for a cutoff ⱕ6, and 0.65 (CI, 0.56 to 0.75) for a cutoff
ⱕ7. RGS ⫽revised Geneva score.
Appendix Figure 1. Receiver-operating characteristic curves of
the 4 clinical decision rules.
Sensitivity
1 – Specificity
No discrimination
Wells rule
Simplified Wells rule
Revised Geneva score
Simplified revised Geneva score
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Area under the receiver-operating characteristic curves were 0.73 (95%
CI, 0.69 to 0.77) for the Wells rule, 0.72 (CI, 0.68 to 0.76) for the
simplified Wells rule, 0.70 (CI 0.65 to 0.74) for the revised Geneva
score, and 0.69 (CI, 0.65 to 0.74) for the simplified revised Geneva
score.
Appendix Table 3. Discordance Among the Categorization of Unlikely and Likely Clinical Probability According to 4 CDRs in 807
Patients With Suspected PE*
Variable Wells Likely (
n
ⴝ223) SW Likely (
n
ⴝ308) RGS Likely (
n
ⴝ254) SRGS Likely (
n
ⴝ231)
Patients,
n
Patients With
PE,
n
Patients,
n
Patients With
PE,
n
Patients,
n
Patients With
PE,
n
Patients,
n
Patients With
PE,
n
Wells unlikely (n⫽584) 85 25 115 26 100 23
SW unlikely (n⫽499) 0 0 65 14 51 11
RGS unlikely (n⫽553) 84 24 119 37 1 1
SRGS unlikely (n⫽576) 92 28 128 41 24 8
CDR ⫽clinical decision rule; PE ⫽pulmonary embolism; RGS ⫽revised Geneva score; SRGS ⫽simplified revised Geneva score; SW ⫽simplified Wells rule; Wells ⫽
original Wells rule.
*In total, 243 patients had discordant results. The number of patients with discordant CDR results when 2 CDRs are compared can be calculated by adding the number
of patients with an unlikely score according to 1 CDR, but a likely score according to the other CDR, with the number of patients with a likely score according to the first
CDR but an unlikely score according to the second CDR. For instance, to find the number of patients with discordant results when comparing the RGS with the simplified
RGS, use this example: There are 24 patients with a likely RGS result who have an unlikely simplified RGS result. Also, there is 1 patient with an unlikely RGS result but
a likely simplified RGS result. This means there is a total of 25 patients (24 ⫹1⫽25) with discordances when the RGS and simplified RGS are compared, out of a total
of 807 patients (3.1%).
W-252 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 www.annals.org
Appendix Table 4. Frequency Table Using Different Cutoff
Points for the Dichotomized Revised Geneva Score
Cutoff Category Patients, % PE, %
ⱕ4 Unlikely 55 7.9
ⱕ4 Likely 45 26
ⱕ5 Unlikely 70 10
ⱕ5 Likely 30 32
ⱕ6 Unlikely 80 11
ⱕ6 Likely 20 37
ⱕ7 Unlikely 87 12
ⱕ7 Likely 13 48
PE ⫽pulmonary embolism.
www.annals.org 7 June 2011 Annals of Internal Medicine Volume 154 • Number 11 W-253
