External validation of the San Francisco Syncope Rule. Ann Emerg Med 49(4):420-427, 427 e1-4

Abstract

We externally validate the ability of the San Francisco Syncope Rule to accurately identify syncope patients who will experience a 7-day serious clinical event.
Patients who presented to a single academic emergency department (ED) between 8 am and 10 pm with syncope or near-syncope were prospectively enrolled. Treating physicians recorded the presence or absence of all San Francisco Syncope Rule risk factors. Patients were contacted by telephone at 14 days for a structured interview. A 3-physician panel, blinded to the San Francisco Syncope Rule score, reviewed ED medical records, hospital records, and telephone interview forms to identify predefined serious clinical events. The primary outcome was the ability of the San Francisco Syncope Rule to predict any 7-day serious clinical event. A secondary outcome was the ability of the San Francisco Syncope Rule to predict 7-day serious clinical events that were not identified during the initial ED evaluation.
Of 592 eligible patients, 477 (81%) provided informed consent. Direct telephone contact or admission/outpatient records were successfully obtained for 463 (97%) patients. There were 56 (12%) patients who had a serious 7-day clinical event, including 16 (3%) who received a diagnosis after the initial ED evaluation. Sensitivity and specificity of the San Francisco Syncope Rule for the primary outcome were 89% (95% confidence interval [CI] 81% to 97%) and 42% (95% CI 37% to 48%), respectively, and 69% (95% CI 46% to 92%) and 42% (95% CI 37% to 48%), respectively, for the secondary outcome. Estimates of sensitivity were minimally affected by missing data and most optimistic assumptions for missing follow-up information.
In this external validation cohort, the San Francisco Syncope Rule had a lower sensitivity and specificity than in previous reports.

Full text

NEUROLOGY/ORIGINAL RESEARCH
External Validation of the San Francisco Syncope Rule
Benjamin C. Sun, MD, MPP
Carol M. Mangione, MD, MSPH
Guy Merchant, BA
Timothy Weiss, BA
Gil Z. Shlamovitz, MD
Gelareh Zargaraff, MD
Sharon Shiraga, BA
Jerome R. Hoffman, MD, MA
William R. Mower, MD, PhD
From the Department of Medicine, West Los Angeles Veterans Affairs Medical Center,
Los Angeles, CA (Sun); the Department of Medicine (Sun, Mangione), School of Public Health
(Mangione), and the Emergency Medicine Center (Merchant, Weiss, Shlamovitz, Zargaraff,
Shiraga, Hoffman, Mower), University of California at Los Angeles, Los Angeles, CA.
Study objective: We externally validate the ability of the San Francisco Syncope Rule to accurately
identify syncope patients who will experience a 7-day serious clinical event.
Methods: Patients who presented to a single academic emergency department (ED) between 8 AM
and 10 PM with syncope or near-syncope were prospectively enrolled. Treating physicians recorded
the presence or absence of all San Francisco Syncope Rule risk factors. Patients were contacted by
telephone at 14 days for a structured interview. A 3-physician panel, blinded to the San Francisco
Syncope Rule score, reviewed ED medical records, hospital records, and telephone interview forms
to identify predefined serious clinical events. The primary outcome was the ability of the San
Francisco Syncope Rule to predict any 7-day serious clinical event. A secondary outcome was the
ability of the San Francisco Syncope Rule to predict 7-day serious clinical events that were not
identified during the initial ED evaluation.
Results: Of 592 eligible patients, 477 (81%) provided informed consent. Direct telephone contact or
admission/outpatient records were successfully obtained for 463 (97%) patients. There were 56
(12%) patients who had a serious 7-day clinical event, including 16 (3%) who received a diagnosis
after the initial ED evaluation. Sensitivity and specificity of the San Francisco Syncope Rule for the
primary outcome were 89% (95% confidence interval [CI] 81% to 97%) and 42% (95% CI 37% to
48%), respectively, and 69% (95% CI 46% to 92%) and 42% (95% CI 37% to 48%), respectively, for
the secondary outcome. Estimates of sensitivity were minimally affected by missing data and most
optimistic assumptions for missing follow-up information.
Conclusion: In this external validation cohort, the San Francisco Syncope Rule had a lower sensitivity
and specificity than in previous reports. [Ann Emerg Med. 2007;49:420-427.]
0196-0644/$-see front matter
Copyright ©2007 by the American College of Emergency Physicians.
doi:10.1016/j.annemergmed.2006.11.012
INTRODUCTION
Background and Importance
Syncope, defined as a transient loss of consciousness, may
herald life-threatening events. As a result, patients presenting with
syncope to emergency departments (EDs) are frequently
hospitalized. Syncope accounts for 1% to 3% of all ED visits and
hospital admissions from the ED.
1-7
Unfortunately, current
admission practices result in marginal diagnostic and therapeutic
benefit
8,9
and consume enormous health care resources. Between
39% and 50% of admitted patients are discharged without an
explanation for syncope,
4,10
and syncope-related admissions in the
United States account for $2.4 billion in annual health care costs.
11
The San Francisco Syncope Rule is an instrument designed
to identify patients with syncope or near-syncope who are at low
risk of a short-term, serious clinical event.
12,13
Low-risk patients
can potentially be discharged for an outpatient evaluation of
syncope. The San Francisco Syncope Rule predictors include an
abnormal ECG result, complaint of shortness of breath,
hematocrit level less than 30%, systolic blood pressure less than
90 mm Hg, and a history of congestive heart failure.
420 Annals of Emergency Medicine Volume , .  : April 

In a derivation study, outcomes included any predefined
clinical events within 7 days; the San Francisco Syncope Rule
demonstrated 96% sensitivity (95% confidence interval [CI]
92% to 100%) and 62% specificity (95% CI 58% to 66%).
12
In a validation study, outcomes included 30-day predefined
events that were diagnosed after the ED visit; the San Francisco
Syncope Rule demonstrated 98% sensitivity (95% CI 89% to
100%) and 56% specificity (95% CI 52% to 60%).
13
These
studies suggest that application of the San Francisco Syncope
Rule may safely decrease syncope-related admissions by 7%
to 10%.
Goals of This Investigation
The published San Francisco Syncope Rule derivation and
validation studies have been performed at the same institution.
Our goal was to evaluate the accuracy of the San Francisco
Syncope Rule to identify “low-risk” patients in an independent,
prospective validation sample. The primary outcome included
all 7-day serious clinical events. The secondary outcome
included 7-day serious clinical events that were diagnosed only
after the index ED visit. Analysis of the secondary outcome is
important because there is little clinical utility in “predicting”
serious conditions that are evident at the ED evaluation.
MATERIALS AND METHODS
Study Design and Setting
This was a single-center, prospective, observational, cohort
study that enrolled patients from April 18, 2005, to April 18,
2006. The study site is an urban, academic, Level I trauma
center with an emergency medicine residency and an annual
volume of 40,000 visits. The study site institutional review
board approved the research protocol.
Selection of Participants
Adult patients with a complaint of syncope or near-syncope
were eligible for enrollment. Syncope is defined as a sudden,
transient loss of consciousness. Near-syncope is defined as a
sensation of imminent loss of consciousness, without actual
syncope. The treating resident or attending physician
determined patient eligibility for study enrollment.
Exclusion criteria included loss of consciousness related to a
witnessed seizure, loss of consciousness after head trauma,
ongoing confusion (including baseline cognitive impairment or
dementia), intoxication, age younger than 18 years, inability to
speak English or Spanish, do-not-resuscitate (DNR) or do-not-
intubate (DNI) status, and lack of follow-up contact
information.
An ED-based research assistant was available from 8 AM to
10 PM, 7 days a week. Research assistants identified all
potentially eligible patients by reviewing the ED intake log and
querying the charge nurse, attending physicians, and resident
physicians as they were evaluating active ED patients. A research
assistant explained the goals of the study to eligible patients and
obtained informed consent for enrollment. Retrospective
internal quality checks, including medical record review and ED
intake log review, demonstrated that 76% of eligible patients
were identified and screened. There were no differences in age
and sex among potentially eligible patients who were screened
and those who were not screened.
All elements of the San Francisco Syncope Rule were
collected prospectively. For each enrolled patient, the treating
resident emergency physician completed a structured data form
about that patient’s clinical presentation (Appendix E1 available
online at http://www.annemergmed.com). Presence of shortness
of breath, history of congestive heart failure, or abnormal ECG
result was obtained from the physician structured data form.
For shortness of breath and history of congestive heart failure,
the treating physician could record “yes,” “no,” or “unknown.”
The ECG was considered abnormal if the treating physician
noted it to show any rhythm other than sinus, any bundle-
branch block, left-axis deviation, mono- or biventricular
hypertrophy, any abnormal conduction interval except for first-
degree atrioventricular block, any Q/ST/T change consistent
with ischemia (acute or chronic), or isolated, nonspecific ST/T
abnormalities. A research assistant verified completeness of data
forms and abstracted triage systolic blood pressure from nursing
notes. Hematocrit values were obtained from a computerized
laboratory reporting system. Information about age, sex, race,
and ethnicity were obtained from administrative data.
The clinical evaluation data forms were completed by
emergency medicine residents with 2 to 4 years of experience.
To assess the interrater reliability of clinical predictors and to
assess San Francisco Syncope Rule accuracy as a function of
Editor’s Capsule Summary
What is already known on this topic
The San Francisco Syncope Rule was developed to
identify emergency department (ED) syncope patients
who are at low risk for adverse events within the
succeeding 7 days. In initial derivation and validation
studies, the San Francisco Syncope Rule was found to
have a sensitivity of 96% to 98%.
What question this study addressed
Does the San Francisco Syncope Rule perform
comparably in a similar sample of patients from a
different ED?
What this study adds to our knowledge
The sensitivity of the San Francisco Syncope Rule was
89%. It is unclear whether the lower sensitivity represents
different performance of the rule, different application of
the rule, or random variation.
How this might change clinical practice
These results suggest that the San Francisco Syncope
Rule requires further external validation before being
adopted into general ED use.
Sun et al External Validation of San Francisco Syncope Rule
Volume , .  : April  Annals of Emergency Medicine 421

clinician experience, the attending emergency physician
independently completed a second data form in a convenience
sample of patients. The research assistant requested that the
attending physician complete a second form for all patients;
however, attending physicians were occasionally unable to
complete the form because of other clinical responsibilities.
Attending and resident physicians were trained in the
completion of the data forms in 1 session before and 1 session
after initiation of the study. Physicians were instructed to treat
and admit patients in their usual manner, without any specific
study intervention or testing.
For patients who did not receive ECG testing as part of
routine care, a research assistant obtained patient permission to
perform a study ECG. Study ECGs were immediately sealed,
were not available to treating physicians, and were interpreted
later by a study investigator (B.C.S.) who was blinded to other
information about the patient’s presentation. Hematocrit testing
was performed at the discretion of treating physicians, and we
did not collect hematocrit data if this test was not ordered as
part of routine care.
The primary outcome included all 7-day predefined serious
clinical events. The secondary outcome included 7-day serious
clinical events that were diagnosed only after the initial ED
evaluation.
Predefined outcomes included death, myocardial infarction,
arrhythmias, pulmonary embolism, stroke or transient ischemic
attack, subarachnoid or nontraumatic cerebral hemorrhage,
aortic dissection, new diagnosis of structural heart disease
thought to be related to syncope, and significant hemorrhage or
anemia requiring blood transfusion. Explicit criteria were used
to identify outcomes (Appendix E2, available online at http://
www.annemergmed.com). Any patient who was discharged
from the ED or hospital after an episode of syncope and then
readmitted for similar symptoms related to the initial event was
considered to have a serious outcome. Admitted patients who
required a predefined acute intervention during their stay were
also considered to have a serious outcome. An acute
intervention was defined as a procedure required to treat a
condition related to a patient’s symptoms of syncope, including
pacemaker insertion, coronary angioplasty, surgery for valvular
heart disease, balloon pump insertion, use of vasopressors,
surgery to treat an abdominal aortic aneurysm surgery for
ruptured spleen, surgery for ruptured ectopic pregnancy, and
endoscopic treatment of esophageal varices. Our outcomes
definitions are consistent with those used by the San Francisco
Syncope Rule investigators,
12,13
and the outcomes period is
consistent with that of the original derivation study.
12
Outcomes Measurement
Direct patient telephone follow-up was performed to identify
admissions or serious clinical events that occurred outside the
study site. We attempted to contact all patients at 14 days after
the index ED visit for a structured telephone interview by a
research assistant (Appendix E3, available online at
http://www.annemergmed.com). Inpatient records and
discharge summaries were obtained for all patients transferred
from the study site ED to other hospitals for admission.
Two emergency physicians independently reviewed available
ED documentation, inpatient records, and telephone interview
forms for all enrolled patients. Records for all patients identified
as potentially experiencing a serious outcome were then
reviewed by a panel of 3 emergency physicians. All reviewing
physicians were blinded to the structured data forms completed
by treating physicians. Occurrence and timing of the
predefined, serious clinical events were determined through
panel consensus and recorded on a structured data form
(Appendix E4, available online at http://www.
annemergmed.com).
According to published data, we assumed that the San
Francisco Syncope Rule would have a sensitivity of 98%.
12,13
Power analyses suggested that approximately 50 enrolled
patients with the primary outcome would be required to achieve
a 95% CI width of 10%.
Primary Data Analysis
We performed descriptive analysis of baseline patient
characteristics stratified by occurrence of a serious outcome.
Between-group differences and CIs in characteristics were
estimated using Wilson estimates for binomial variables. The
␬
statistic was used to assess the interrater reliability of clinical
predictors.
In the base case analysis, we analyzed all patients who either
had the presence of at least 1 of the 5 high-risk predictors
(because these patients were classified as “at risk” regardless of
missing data on other predictors) or who had complete data on
all 5 predictors recorded on the form (completed by a resident
physician). For shortness of breath and history of congestive
heart failure, we treated a response of “unknown” as missing
data. ECG data were considered missing if an ECG was not
ordered by the treating physician and a study ECG was not
performed. Hematocrit data were considered missing if this test
was not ordered by the treating physician. We determined
sensitivity, specificity, negative predictive value, and positive
predictive value of the San Francisco Syncope Rule to identify
patients with all 7-day outcomes, as well as 7-day outcomes that
were diagnosed after the initial ED evaluation. In this base case
subgroup, we compared the San Francisco Syncope Rule test
characteristics with physician judgment, which we defined as
the treating physicians’ decision to hospitalize the patient.
Sensitivity Analyses
We performed sensitivity analyses to assess the impact of
missing data. We estimated the upper bounds of sensitivity by
assuming that all missing data represented the presence of a
predictor. To estimate the upper bounds of specificity, we
assumed that all missing data represented the absence of a
predictor. We compared the results of these sensitivity analyses
against the treating physician’s judgment in the entire study
cohort.
External Validation of San Francisco Syncope Rule Sun et al
422 Annals of Emergency Medicine Volume , .  : April 

We also assessed the potential impact of missing follow-up
information, again in a manner that would estimate the maximal
sensitivity and specificity of the San Francisco Syncope Rule. We
made the following assumptions for patients without follow-up
information: (1) for patients with at least 1 San Francisco Syncope
Rule high-risk predictor or who had complete data on all San
Francisco Syncope Rule predictors, we assumed that the patient
experienced the outcome predicted by the San Francisco Syncope
Rule; and (2) for patients with missing data on the San Francisco
Syncope Rule predictors and who did not have at least 1 San
Francisco Syncope Rule high-risk predictor, we assumed that the
patient was classified as low risk by the San Francisco Syncope Rule
and did not experience any serious clinical event. In a separate
analysis to estimate the lower bounds of sensitivity and specificity
caused by missing follow-up information, we made the following
assumption for patients without follow-up: (1) for patients with at
least 1 San Francisco Syncope Rule predictor, we assumed that the
patient did not experience a serious outcome; and (2) for patients
who did not have any of the San Francisco Syncope Rule
predictors, we assumed that the patient did experience a serious
outcome.
Finally, we assessed whether San Francisco Syncope Rule test
characteristics were dependent on the clinical experience of the
treating physician. We repeated the analysis described above
using clinical evaluation data completed by attending
physicians.
Data management and statistical analyses were conducted
using SAS software, version 8.02 (SAS Institute Inc., Cary,
NC). Test characteristics and associated 95% CIs were
calculated using a publicly available SAS macro.
14
RESULTS
Of the 709 patients who were screened during the study
period, 592 (83%) were eligible, and 477 (81%) provided
informed consent to participate. We found no important
differences in age, sex, race, or ethnicity between eligible
patients who provided or declined informed consent. Direct
telephone follow-up was obtained in 436 patients (91%). Of the
remaining 41 patients, 27 (6% of the entire cohort) had
available inpatient or outpatient data for at least 2 weeks after
the date of enrollment, and 14 (3%) had no available follow-up
information. Patients without any available follow-up
information were younger and more likely to be of nonwhite
race compared with patients with follow-up information.
Study sample characteristics are presented in Table 1. Of the
overall cohort, 58% were admitted or transferred to another
hospital. Blood pressure data were available for all patients, and
data were missing in less than 1.5% of patients for presence of
Table 1. Study population characteristics.*
Characteristic
All Patients
(nⴝ477), %
No Serious Event
(nⴝ421), %
7-Day Serious Event
(nⴝ56), %
Difference Between Groups
(95% CI), %
†
Age, median, y (IQR) 58 (35, 79) 55 (32, 78) 76 (60, 85)
⬍40 30 33 7 24 (9 to 40)
40-59 23 24 18 5 (⫺7to17)
60-79 24 22 38 ⫺15 (⫺26 to ⫺5)
⬎80 21 21 38 ⫺17 (⫺27 to ⫺7)
Male 44 41 66 ⫺24 (⫺35 to ⫺13)
Hispanic 10 10 11 ⫺2(⫺10 to 6)
Race
White 77 76 86 ⫺8(⫺18 to 1)
Black 9 10 4 5 (⫺4to13)
Asian 11 11 11 ⫺1(⫺10 to 7)
Other 3 3 0 2 (⫺3to7)
Disposition
Admitted 51 45 93 ⫺46 (⫺54 to ⫺39)
Transferred 7 7 7 ⫺1(⫺8to6)
Discharged 40 45 0 44 (25 to 62)
Left against medical advice 2 2 0 1 (⫺4to5)
SFSR predictors
Abnormal ECG 37 32 73 ⫺41 (⫺50 to ⫺32)
Shortness of breath 12 11 20 ⫺10 (⫺18 to ⫺2)
Hematocrit ⬍30 6 4 15 ⫺12 (⫺17 to ⫺8)
SBP ⬍90 mm Hg 5 5 9 ⫺5(⫺11 to 1)
History of CHF 8 7 16 ⫺10 (⫺17 to ⫺3)
Any SFSR predictor 46 41 89 ⫺46 (⫺53 to ⫺38)
IQR, Interquartile range; SBP, systolic blood pressure; CHF, congestive heart failure; SFSR, San Francisco Syncope Rule.
*Subgroups may not add to exactly 100% because values are rounded to the nearest integer.
†
Represents group difference of (no serious event)–(serious event); point estimates are based on Wilson estimates and may not be exactly the same as the absolute
difference of group proportions.
Sun et al External Validation of San Francisco Syncope Rule
Volume , .  : April  Annals of Emergency Medicine 423

shortness of breath (n⫽7) or history of congestive heart failure
(n⫽6). Less than 7% of patients did not receive an ECG as part
of routine care and refused a study ECG (n⫽33). Finally, 32%
of patients did not receive hematocrit testing as part of routine
care (n⫽153).
The proportion of patients with any 7-day serious clinical
event was 11.7% (56 of 477 patients), and the proportion of
patients with a 7-day serious clinical event that was diagnosed
only after the index ED visit was 3.4% (16 of 477 patients).
Serious clinical events are described in Table 2.
Second rater evaluation forms were completed by an
attending physician in a convenience sample of 307 patients.
Interrater reliability for presence of shortness of breath,
abnormal ECG, and presence of any San Francisco Syncope
Rule predictor was modest (Table 3).
Table 4 presents the test characteristics of the San Francisco
Syncope Rule under various assumptions. In the base case
analysis, a decision by the managing physicians to admit the
patient was 100% sensitive and 30% specific for all 7-day
outcomes. The point estimates for San Francisco Syncope Rule
sensitivity were 89% for any 7-day serious event and 69% for a
7-day serious event that was diagnosed only after the initial ED
visit. If patients without San Francisco Syncope Rule risk factors
were discharged and all others admitted, hospitalizations would
have decreased in the base case subgroup by 12%, but at the
price of having discharged 5 patients (10% of primary
outcomes) who would have subsequently been diagnosed with a
serious event.
Estimates of sensitivity were minimally affected by missing
data on the San Francisco Syncope Rule predictors, whereas
estimates of specificity depended on how missing data were
handled. San Francisco Syncope Rule test characteristics were
minimally affected by missing follow-up information when the
most optimistic assumptions were used. Point estimates for
sensitivity and specificity were lower when elements of the San
Francisco Syncope Rule were calculated from forms completed
by an attending physician. The Figure describes patients who
were misclassified as “low risk” by the San Francisco Syncope
Rule in the base case analysis.
LIMITATIONS
We performed a prospective cohort study designed to
minimize missing data bias and maximize direct patient follow-
up with a standardized protocol. Nevertheless, there are
potential limitations to our study.
We excluded children, patients with DNR/DNI status, and
patients without follow-up contact information, whereas these
were not exclusion criteria used by the San Francisco Syncope
Rule investigators. It is possible that the San Francisco Syncope
Rule demonstrated different test characteristics in these patients.
We did not enroll patients between 10 PM and 8 AM, for
logistic reasons, and this may be a source of selection bias. We
did not detect age or sex differences between eligible patients
during study hours and patients with a chief complaint of
“syncope” who presented after study hours, although we did not
have data to perform a more detailed analysis of differences
between the 2 groups. However, we have no reason to believe
that the San Francisco Syncope Rule would have different test
characteristics in patients who present between 10 PM and 8 AM.
We successfully screened 76% of potentially eligible patients
during study hours. Most of the potentially eligible patients
who were not screened were missed because of research assistant
unavailability (eg, multiple patients requiring screening at the
same time, or gaps in research assistant coverage schedule).
Although we did not find a difference in age and sex between
potentially eligible patients who were screened and those who
were not, this may be a source of selection bias.
We collected data on triage blood pressure and did not have
data on other recorded blood pressure measurements during the
ED visit. Clinicians may be influenced by the lowest measured
blood pressure value, and future studies should explore the
importance of serial blood pressure measurements.
ECG and hematocrit testing were not available in 7% and
32% of enrolled patients, respectively. Of patients with a 7-day
serious outcome, all but 1 patient received an ECG, and all
received hematocrit testing. A 21-year-old man diagnosed with
upper gastrointestinal bleeding and a hematocrit level of 22%
did not receive ECG testing. Thus, such missing data could not
substantially affect our estimates of sensitivity, whereas it tends
to provide an upwardly biased estimate of specificity because of
Table 2. Seven-day serious clinical events.*
Type of Event
†
All 7-Day
Events No. (%)
“Delayed” 7-Day
Events No. (%)
Arrhythmia 32 (57) 7 (44)
Hemorrhage/anemia 8 (14) 0
Structural heart disease 6 (11) 3 (19)
Stroke/TIA 2 (4) 2 (13)
Pacemaker placement 2 (4) 2 (13)
PTCA 2 (4) 2 (13)
Myocardial infarction 1 (2) 0
Pulmonary embolism 1 (2) 0
Cerebral hemorrhage 1 (2) 0
Aortic dissection 1 (2) 0
Total 56 16
*Some patients experienced more than 1 event; this table classifies patients by
the first clinically observed event.
†
Denominator is total number of patients experiencing an event; columns may
not add to exactly 100% because values are rounded to the nearest integer.
Table 3. Interobserver agreement for shortness of breath and
abnormal ECG.
Predictor
Subgroup
No.
Percentage
Agreement
␬
(95% CI)
Shortness of breath 288 90 0.5 (0.3–0.6)
Abnormal ECG 241 75 0.5 (0.4–0.6)
Any high-risk predictor* 225 85 0.6 (0.5–0.7)
*Observations were dropped if either rater did not complete all elements of the
SFSR and no abnormal findings were present.
External Validation of San Francisco Syncope Rule Sun et al
424 Annals of Emergency Medicine Volume , .  : April 

the potential for false-positive test results, which is supported by
results of our preplanned sensitivity analysis assessing the effects
of missing data.
Our definition of an abnormal ECG was slightly different
from that used by the San Francisco Syncope Rule
investigators. The San Francisco Syncope Rule derivation
study considered an ECG with new changes to be abnormal.
We did not have uniform data on past ECGs on enrolled
patients; therefore, any predefined findings on ECG were
considered abnormal, regardless of past ECG testing.
Therefore, our estimates of sensitivity are likely to be biased
upward and specificity downward because of this
measurement difference, although the effect size of this
variation is likely minimal.
We used outcomes as defined by the original San Francisco
Syncope Rule derivation study, including all 7-day predefined,
serious clinical events.
12
In their validation study, however, the San
Francisco Syncope Rule investigators used a different outcomes
definition, which included 30-day serious clinical events that were
diagnosed after the initial ED presentation.
13
In pilot work, we
found an unacceptable decrease in telephone interview response
rate when the follow-up period was extended from 14 to 30 days.
Therefore, we cannot assess the exact outcome used in the
validation study reported by the San Francisco Syncope Rule
validation study. In post hoc analysis to assess the impact of
measuring a longer period, however, we found no qualitative
differences in San Francisco Syncope Rule sensitivity estimates
when we used a 14-day period for all serious events, as well as
events that were diagnosed after the initial ED visit.
This was a single-center study, and it is possible that the San
Francisco Syncope Rule demonstrates different test
characteristics when applied by physicians at other institutions.
Finally, we observed a relatively small number of serious events
in our study cohort, and a larger study with more outcomes
would provide narrower 95% CIs for estimates of San Francisco
Syncope Rule sensitivity and specificity.
DISCUSSION
We performed a prospective cohort study to externally
validate the San Francisco Syncope Rule. Study strengths
Figure. Patients misclassified as “low risk” in the base
case analysis.
Table 4. San Francisco Syncope Rule test characteristics with likelihood ratios.
Assumption No.
Sensitivity
(95% CI)
Specificity
(95% CI)
Positive
Predictive
Value (95% CI)
Negative
Predictive
Value (95% CI)
Likelihood
Ratio
Positive
Likelihood
Ratio
Negative
All 7-day serious outcomes
Physician decision to admit, base case* 351 100 (94–100) 30 (25–35) 21 (16–26) 100 (96–100) 1.4 0
SFSR base case* 351 89 (81–97) 42 (37–48) 22 (17–28) 95 (92–99) 1.5 0.3
Physician decision to admit, all patients 477 100 (94–100) 47 (42–51) 20 (19–25) 100 (98–100) 1.9 0
SFSR missing predictors set to “present” 477 89 (81–97) 30 (25–34) 15 (11–18) 95 (92–99) 1.3 0.4
SFSR missing predictors set to “absent” 477 88 (79–96) 59 (55–64) 22 (17–28) 97 (95–99) 2.1 0.2
“Delayed” 7-day serious outcomes (exclude 40 patients with serious events diagnosed during ED evaluation)
SFSR base case* 311 69 (46–92) 42 (37–48) 6 (3–10) 96 (93–100) 1.2 0.7
SFSR missing predictors set to “present” 437 69 (46–92) 30 (25–34) 4 (2–6) 96 (93–99) 1.0 1.0
SFSR missing predictors set to “absent” 437 69 (46–92) 59 (55–64) 6 (3–10) 98 (96–100) 1.7 0.5
All 7-day serious outcomes, best case assumptions for missing follow-up information
†
SFSR base case* 361 90 (83–98) 45 (40–51) 26 (20–32) 96 (92–99) 1.6 0.2
All 7-day serious outcomes, worst case assumptions for missing follow-up information
†
SFSR base case* 361 75 (63–84) 42 (36–48) 22 (17–29) 88 (81–93) 1.3 0.6
All 7-day serious outcomes, resident and attending evaluations available
SFSR base case, attending physicians* 206 81 (69–94) 32 (25–39) 20 (14–27) 89 (81–97) 1.2 0.6
SFSR base case, residents* 206 89 (79–99) 40 (32–47) 25 (17–32) 94 (89,100) 1.5 0.3
*Includes patients for whom an SFSR score could be calculated.
†
See Materials and Methods section for description of assumptions.
Sun et al External Validation of San Francisco Syncope Rule
Volume , .  : April  Annals of Emergency Medicine 425

include a protocol to minimize missing ECG data, high rates of
direct patient follow-up, a physician outcomes review panel that
was blinded to data recorded by treating physicians, and explicit
methods for assessing the effects of missing data on San
Francisco Syncope Rule predictors and missing follow-up
information. At our institution, physician application of the San
Francisco Syncope Rule demonstrated lower sensitivity and
specificity than previously reported.
12,13
In particular, the San
Francisco Syncope Rule demonstrated poor sensitivity for
identifying patients with a serious event that was first diagnosed
after the initial ED visit. Estimates of sensitivity were minimally
affected by missing ECG and hematocrit data, whereas estimates
of specificity depended on how missing data were analyzed. Test
characteristics were minimally affected by missing follow-up
information under the most optimistic assumptions. Treating
physicians admitted all patients who developed a 7-day serious
event. Application of the San Francisco Syncope Rule in this
study cohort could have reduced hospitalizations by 12%, but at
the price of missing 10% of patients with a 7-day serious
outcome.
Two previous syncope risk prediction instruments focused
on 1-year outcomes, including death or arrhythmias.
15,16
Risk
prediction for 1-year outcomes, however, is unlikely to be
helpful for ED decisionmaking about whether to acutely
hospitalize a patient. A recent study reported a risk score for
predicting arrhythmias in patients with unexplained syncope,
although the study entry criteria required research physician
evaluation and did not specify when outcomes occurred.
17
The San Francisco Syncope Rule represents a potentially
important advance in syncope risk stratification for emergency
physicians because of its focus on short-term serious events for
which hospitalization may be beneficial. The San Francisco
Syncope Rule investigators have reported the results of a
derivation and validation study. The derivation study included
684 enrolled patients, and 79 (11.5%) experienced a 7-day
serious event. The San Francisco Syncope Rule demonstrated
96% sensitivity (95% CI 92% to 100%) and 62% specificity
(95% CI 58% to 66%). The proportion of patients with a
serious event that was diagnosed after the initial ED evaluation
was not reported for the derivation study. The validation study
included 791 patients, and 54 (6.8%) experienced a 30-day
serious event after their initial ED visit. The San Francisco
Syncope Rule demonstrated 98% sensitivity (95% CI 89% to
100%) and 56% specificity (95% CI 52% to 60%) in the
validation study. ECG and hematocrit testing, 2 of the San
Francisco Syncope Rule predictors, were not mandatory in
either the San Francisco Syncope Rule derivation or validation
study, and it is unclear how frequently these tests were ordered
and how missing data affected the reported results.
The lower sensitivity and specificity of the San Francisco
Syncope Rule noted in our study may have several explanations.
First, physicians at our site may be unskilled in applying the San
Francisco Syncope Rule criteria. The criteria are relatively
simple, however, and we did provide 2 training sessions to train
study site physicians in the completion of the data form. We did
not find that the San Francisco Syncope Rule test characteristics
improved when applied by attending physicians compared to
resident physicians. Second, it is possible that San Francisco
Syncope Rule accuracy varies by patient population. Our study
site is an academic, tertiary-care referral center that treats many
patients with complex medical conditions. However, the San
Francisco Syncope Rule was derived and validated at a center
with similar characteristics. Third, because of the relatively small
number of patients who experienced a serious event in our
study, there are fairly wide CIs around our point estimates for
sensitivity and specificity. CIs for sensitivity overlap from our
study and the San Francisco Syncope Rule derivation and
validation reports. However, our findings suggest that the San
Francisco Syncope Rule may require further validation before
safe clinical application.
We propose the following recommendations for future
syncope risk stratification research. First, research should focus
on short-term serious events that are diagnosed after the initial
ED visit. In our cohort, internal hemorrhage and anemia were
always diagnosed in the ED, whereas undeclared outcomes
included only cardiac and neurologic events. Narrowing the
scope of outcomes may result in a more accurate and clinically
intuitive prediction instrument.
Second, future studies should include sufficient numbers of
serious events to allow for robust multivariate risk prediction
analysis. Because the rate of delayed serious events is low
(approximately 3% at 7 days in our study), an adequately
powered study may require thousands of patients.
Finally, older patients (age ⬎60 years) accounted for the
majority of serious events in our cohort, as well as the majority
of serious outcomes missed by the San Francisco Syncope Rule.
Although other investigators have found age to be a powerful
predictor of syncope-related adverse events,
15-20
the San
Francisco Syncope Rule does not include age as a risk factor.
The importance of age as a risk predictor should be reevaluated
in future studies. An ideal study setting for further risk
stratification research would be a large, population-based,
closed-cohort system, with older patients accounting for a high
proportion of syncope-related ED visits.
In summary, we performed an external validation study of
the San Francisco Syncope Rule. In our cohort, the San
Francisco Syncope Rule displayed lower sensitivity and
specificity for 7-day serious events than previously reported.
Point estimates of sensitivity were minimally affected by missing
data. Test characteristics were minimally affected by missing
follow-up information under the most optimistic assumptions.
Our results suggest that further validation studies should be
performed before widespread implementation of the San
Francisco Syncope Rule.
We are indebted to the efforts of the University of California, Los
Angeles emergency medicine research associates who performed
patient enrollment, data entry, and telephone follow-up interviews.
External Validation of San Francisco Syncope Rule Sun et al
426 Annals of Emergency Medicine Volume , .  : April 

We also thank the residents, faculty, and nursing staff who cared for
these patients and completed the data forms.
Supervising editor: Allan B. Wolfson, MD
Author contributions: BCS, CMM, JRH, and WRM conceived
the study. BCS and CMM obtained funding for this study.
BCS, GM, TW, GZS, GZ, and SS were responsible for data
collection, and BCS supervised the overall data collection
process. BCS performed the data analysis and drafted the
article. All authors contributed substantially to article
revisions. BCS takes responsibility for the paper as a whole.
Michael Callaham, MD, recused himself from the editorial
decision process for this article.
Funding and support: This study was supported by the UCLA
Robert Wood Johnson Clinical Scholars Program (050721). Dr
Sun is supported by a UCLA National Institute of Aging K12
award (K12AG001004) and an American Geriatrics Society
Dennis Jahnigen Career Development Award. Dr Mangione
was also partially supported by the UCLA Center for Health
Improvement in Minority Elders/Resource Centers for Minority
Aging Research, NIH/NIA (AG 02-004).
Publication dates: Received for publication September 11,
2006. Revision received October 27, 2006. Accepted for
publication November 10, 2006. Available online January 8,
2007.
Reprints not available from the authors.
Address for correspondence: Benjamin Sun, MD, MPP, Office
3214A, Mail Stop 111, Building 500, Wing 3E, West Los
Angeles Veterans Affairs Medical Center, 11301 Wilshire Blvd,
Los Angeles, CA 90073; 310-268-3125, fax 310-268-4818;
E-mail bsun@post.harvard.edu.
REFERENCES
1. Brignole M, Disertori M, Menozzi C, et al. Management of
syncope referred urgently to general hospitals with and without
syncope units. Europace. 2003;5:293-298.
2. Casini-Raggi V, Bandinelli G, Lagi A. Vasovagal syncope in
emergency room patients: analysis of a metropolitan area
registry. Neuroepidemiology. 2002;21:287-291.
3. Ammirati F, Colivicchi F, Santini M. Diagnosing syncope in clinical
practice: implementation of a simplified diagnostic algorithm in a
multicentre prospective trial: the OESIL 2 study (Osservatorio
Epidemiologico della Sincope nel Lazio). Eur Heart J. 2000;21:
935-940.
4. Blanc JJ, L’Her C, Touiza A, et al. Prospective evaluation and
outcome of patients admitted for syncope over a 1 year period.
Eur Heart J. 2002;23:815-820.
5. Day SC, Cook EF, Funkenstein H, et al. Evaluation and outcome
of emergency room patients with transient loss of consciousness.
Am J Med. 1982;73:15-23.
6. Sarasin FP, Louis-Simonet M, Carballo D, et al. Prospective
evaluation of patients with syncope: a population-based study.
Am J Med. 2001;111:177-184.
7. Sun BC, Emond JA, Camargo CA Jr. Characteristics and
admission patterns of patients presenting with syncope to U.S.
emergency departments, 1992-2000. Acad Emerg Med. 2004;11:
1029-1034.
8. Mozes B, Confino-Cohen R, Halkin H. Cost-effectiveness of in-
hospital evaluation of patients with syncope. Isr J Med Sci. 1988;
24:302-306.
9. Kapoor WN, Karpf M, Maher Y, et al. Syncope of unknown origin:
the need for a more cost-effective approach to its diagnosis
evaluation. JAMA. 1982;247:2687-2691.
10. Getchell WS, Larsen GC, Morris CD, et al. Epidemiology of
syncope in hospitalized patients. J Gen Intern Med. 1999;14:
677-687.
11. Sun BC, Emond JA, Camargo CA Jr. Direct medical costs of
syncope-related hospitalizations in the United States. Am J
Cardiol. 2005;95:668-671.
12. Quinn JV, Stiell IG, McDermott DA, et al. Derivation of the San
Francisco Syncope Rule to predict patients with short-term
serious outcomes. Ann Emerg Med. 2004;43:224-232.
13. Quinn J, McDermott D, Stiell I, et al. Prospective validation of the
San Francisco Syncope Rule to predict patients with serious
outcomes. Ann Emerg Med. 2006;47:448-454.
14. Cha S, Senspe BS. Mayo Clinic Division of Biostatistics. Available
at: http://mayoresearch.mayo.edu/mayo/research/biostat/
sasmacros.cfm. Accessed April 16, 2006.
15. Martin TP, Hanusa BH, Kapoor WN. Risk stratification of patients
with syncope. Ann Emerg Med. 1997;29:459-466.
16. Colivicchi F, Ammirati F, Melina D, et al. Development and
prospective validation of a risk stratification system for patients
with syncope in the emergency department: the OESIL risk score.
Eur Heart J. 2003;24:811-819.
17. Sarasin FP, Hanusa BH, Perneger T, et al. A risk score to predict
arrhythmias in patients with unexplained syncope. Acad Emerg
Med. 2003;10:1312-1317.
18. Lipsitz LA, Wei JY, Rowe JW. Syncope in an elderly,
institutionalised population: prevalence, incidence, and
associated risk. Q J Med. 1985;55:45-54.
19. Kenny RA. Syncope in the elderly: diagnosis, evaluation, and
treatment. J Cardiovasc Electrophysiol. 2003;14(9 suppl):
S74-S77.
20. Kapoor W, Snustad D, Peterson J, et al. Syncope in the elderly.
Am J Med. 1986;80:419-428.
Did you know?
You can save your online searches and get the results by e-mail.
Visit www.annemergmed.com today to see what else is new online!
Sun et al External Validation of San Francisco Syncope Rule
Volume , .  : April  Annals of Emergency Medicine 427

Appendix E1. Data Collection Form*
*This is an abbreviated version of the data collection form that includes variables relevant
for this report.
427.e1 Annals of Emergency Medicine Volume 49, .  : April 

Appendix E2. Explicit Criteria for Outcomes.
Arrhythmia
●Ventricular tachycardia ⬎3 beats
●Sick sinus disease, with alternating sinus bradycardia and tachycardia
●Sinus pause ⬎3s
●Third-degree atrioventricular block
●Mobitz II atrioventricular block
●Symptomatic supraventricular tachycardia (pulse rate ⬎100 beats/min)
●Symptomatic atrial flutter or fibrillation with rapid ventricular response (pulse rate ⬎100 beats/min)
●Symptomatic bradycardia (pulse rate ⬍60 beats/min), OR pulse rate ⬍40.
●“Symptomatic” refers to the simultaneous occurrence of dizziness, lightheadedness, hypotension (systolic blood pressure ⬍90), or
syncope with an arrhythmia on ECG monitoring.
Myocardial Infarction
●Requires increase of troponin or ECG change AND
●Documentation that cardiology consultation concurs with diagnosis of myocardial infarction
Stroke/TIA
●Documentation that neurology consultation concurs with diagnosis of stroke/TIA
Pulmonary Embolism
●Requires confirmatory testing (high-probability VQ scan, chest CTA, pulmonary angiography, or new deep vein thrombosis noted on
duplex ultrasonography WITH an abnormal VQ scan OR abnormal chest CTA OR abnormal pulmonary angiography)
Aortic Dissection
●Requires confirmatory testing (chest CT, transesophageal echocardiogram, MRI, or angiography)
Subarachnoid or Intracranial Hemorrhage (Nontraumatic)
●Requires that neurology or neurosurgical consultation concur with diagnosis
Internal Hemorrhage/Anemia Requiring Transfusion
●Any source of bleeding (GI, vaginal, ruptured AAA) or anemia requiring blood transfusion (includes patients who refuse recommended
transfusion, eg, Jehovah’s witnesses)
Volume 49, .  : April  Annals of Emergency Medicine 427.e2

Appendix E3. Contact script for telephone interview.
427.e3 Annals of Emergency Medicine Volume 49, .  : April 

Appendix E4. Adverse Event Form.
Volume 49, .  : April  Annals of Emergency Medicine 427.e4