Content uploaded by Jennifer C Nelson
Author content
All content in this area was uploaded by Jennifer C Nelson
Content may be subject to copyright.
Published by Oxford University Press on behalf of the International Epidemiological Association International Journal of Epidemiology 2006;35:337–344
Ó The Author 2005; all rights reserved. Advance Access publication 20 December 2005 doi:10.1093/ije/dyi274
Evidence of bias in estimates of influenza
vaccine effectiveness in seniors
Lisa A Jackson,
1
,
2
* Michael L Jackson,
1
,
2
Jennifer C Nelson,
1
,
3
Kathleen M Neuzil
4
and Noel S Weiss
2
Accepted 3 November 2005
Background Numerous observational studies have reported that seniors who receive influenza
vaccine are at substantially lower risk of death and hospitalization during the
influenza season than unvaccinated seniors. These estimates could be influenced
by differences in underlying health status between the vaccinated and
unvaccinated groups. Since a protective effect of vaccination should be specific
to influenza season, evaluation of non-influenza periods could indicate the
possible contribution of bias to the estimates observed during influenza season.
Methods We evaluated a cohort of 72 527 persons 65 years of age and older followed
during an 8 year period and assessed the risk of death from any cause, or
hospitalization for pneumonia or influenza, in relation to influenza vaccination,
in periods before, during, and after influenza seasons. Secondary models
adjusted for covariates defined primarily by diagnosis codes assigned to medical
encounters.
Results The relative risk of death for vaccinated persons compared with unvaccinated
persons was 0.39 [95% confidence interval (95% CI), 0.33–0.47] before influenza
season, 0.56 (0.52–0.61) during influenza season, and 0.74 (0.67–0.80) after
influenza season. The relative risk of pneumonia hospitalization was 0.72
(0.59–0.89) before, 0.82 (0.75–0.89) during, and 0.95 (0.85–1.07) after influenza
season. Adjustment for diagnosis code variables resulted in estimates that were
further from the null, in all time periods.
Conclusions The reductions in risk before influenza season indicate preferential receipt of
vaccine by relatively healthy seniors. Adjustment for diagnosis code variables
did not control for this bias. In this study, the magnitude of the bias demonstrated
by the associations before the influenza season was sufficient to account entirely
for the associations observed during influenza season.
Keywords Influenza/prevention and control, influenza vaccines, cohort studies,
bias(epidemiology), confounding factor, epidemiological
Numerous observational studies have reported that seniors who
receive influenza vaccine are at substantially lower risk of death
and hospitalization during influenza season than unvaccinated
seniors.
1–24
The main issue in interpreting those findings is
whether preferential receipt of vaccine by relatively healthy
seniors could account for some or all of the observed reduction in
the risk of health outcomes. Since influenza is a seasonal
infection, a true protective effect of vaccination should be limited
to periods of influenza viral circulation. Assessment of the
vaccine association during influenza and non-influenza periods
could therefore help to distinguish a true vaccine effect from an
effect of bias due to differences in the underlying characteristics
of the vaccinated and unvaccinated groups.
Several studies have assessed the seasonal specificity of
estimates of influenza vaccine effectiveness by comparing the
vaccine association during influenza season with that during a
later time period, and some of those studies have reported a
reduction in the risk of the non-specific outcomes of death or
hospitalization in vaccinated persons compared with unvaccin-
ated persons during influenza season but not during the later
comparison period. Those findings have been interpreted as
1
Center for Health Studies, Group Health Cooperative, Seattle, WA, USA.
2
Department of Epidemiology, University of Washington, Seattle, WA, USA.
3
Department of Biostatistics, University of Washington, Seattle, WA, USA.
4
Department of Medicine, University of Washington, Sea ttle, WA, USA.
* Corresponding author. Center for Health Studies, MPE-16, 1730 Minor
Avenue, Seattle, WA 98101, USA. E-mail: jackson.l@ghc.org
337
by guest on June 7, 2013http://ije.oxfordjournals.org/Downloaded from
evidence for a true vaccine effect during influenza season.
This interpretation assumes that any influence of bias due to
differences in the underlying characteristics of the vaccinated
and unvaccinated groups is constant over time.
We hypothesized that the magnitude of the underlying
differences that predispose to death and hospitalization between
vaccinated and unvaccinated groups may actually diminish over
time. Our rationale was that if seriously ill seniors are less likely
to receive the influenza vaccine there would be a higher short-
term mortality risk in the unvaccinated group compared with the
vaccinated group. As a consequence of the disproportionately
greater loss of seriously ill persons from the unvaccinated group,
over time, the two groups would become more similar. Other
changes in health status, for better or worse, among members
of both groups would also tend to lead to equilibration of
underlying differences that predispose to death and hospitaliza-
tion over time.
These effects could lead to the finding of a greater reduction in
the relative risk of death or hospitalization in vaccinated persons
compared with unvaccinated persons during influenza season
than in a later comparison period, even in the absence of any
true protective effect of vaccination against influenza infection.
For this reason, evaluations of the seasonal specificity of the
association of influenza vaccination and risk of death and
hospitalization should include a pre-influenza season com-
parison period, when there is almost certainly no true vaccine
effect, and when, unlike the more traditional post-influenza
comparison period, the magnitude of the effect of bias due to
differences in underlying characteristics between the vaccinated
and unvaccinated groups is expected to be at least as strong as
that present during influenza season.
To better assess the possible influence of bias due to
confounding by health status on estimates of influenza vaccine
effectiveness, we followed a large population-based cohort
of seniors from September 1995 through August 2003. We
estimated the relative risks of death, hospitalization for
pneumonia or influenza, and other hospitalization outcomes
in vaccinated versus unvaccinated persons, in periods before,
during, and after influenza season. We also replicated methods
of adjustment for covariates defined by diagnosis codes and
indicators of medical utilization reported by previous influenza
vaccine effectiveness studies to assess their ability to remove
bias due to differences in health status.
Methods
Study population and setting
The study cohort included members of Group Health Cooper-
ative, a health maintenance organization (HMO) in Washington
State with an enrollment of ~350 000 members. Group Health
administrative data systems recorded information on enrollment,
nursing home residence, immunizations, and International
Classification of Diseases, 9th Revision, Clinical Modification
(ICD-9-CM) diagnosis codes assigned to inpatient and out-
patient medical encounters. The cohort for the first study year
of September 1995 through August 1996 consisted of persons
who, as of September 1, 1995, were >65 years of age, had
been enrolled in Group Health for at least 1 year, and were not
residents of a nursing home. Additional Group Health members
newly meeting the eligibility criteria as of September 1 of each
subsequent year through 2002 entered the study cohort on that
date. Study cohort members were followed from their date of
study entry until death, disenrollment from Group Health,
nursing home admission, or the study end date of August 31,
2003, whichever was the earliest.
Outcomes
The primary outcomes were all cause mortality and hospital-
ization with a discharge diagnosis of pneumonia or influenza (PI
hospitalization), defined by ICD-9-CM codes 480 through 487.
Secondary outcomes included hospitalization with a discharge
diagnosis of cerebrovascular disease (ICD-9-CM codes 431–437),
ischaemic heart disease (410–414), congestive heart failure
(428), and injury or trauma (800–904 and 910–959).
Disease covariates
To assess the effect of adjustment for health status covariates
defined according to methods used in previous influenza vaccine
effectiveness studies in HMO populations
11,14
on the associ-
ations of influenza vaccination with risk of death and hospit-
alization, we defined the covariates of heart disease, lung disease,
diabetes mellitus, renal disease, cancer, vasculitis and rheum-
atologic disease, dementia, hypertension, atrial fibrillation, lipid
disorders, hospitalization for pneumonia in the prior year, and
12 or more outpatient visits in the prior year by those methods.
Covariate values were updated on September 1 of each study
year and were based on information recorded in Group Health
administrative data systems in the 12 months prior to that date.
Time periods
We categorized each September through August study year into
periods before, during, and after influenza season based on
estimated dates of the start and end of the influenza season. We
used national influenza viral surveillance data
25–29
to define the
onset and end of each influenza season as the first and last weeks
with at least 50 influenza isolates reported. For comparison, we
also assessed local influenza viral surveillance data reported by
Public Health-Seattle and King County, and defined the onset
and end of influenza season by the first and last occurrences of at
least two consecutive weeks with two or more influenza isolates
reported (Table 1). Since the results of analyses based on either
local or national surveillance data were very similar, unless
otherwise noted, the results presented are based on influenza
periods defined by published national data.
To further differentiate risk over time, we also defined intervals
within the before, during, and after influenza periods. We
defined a ‘high vaccination’ before influenza period as the
interval from the date by which 50% of cohort vaccinations had
been administered (Table 1) to the onset of influenza season.
Within the influenza season, we defined early (the onset of
influenza season to the peak influenza period), peak (the 5 weeks
that spanned the 2 weeks before and after the week of peak viral
circulation), and late (the week after the peak period through
the end of influenza season) influenza periods. We also divided
the after influenza period into a post-influenza period (end of
influenza season through May 31) and summer (June 1 through
August 31).
338 INTERNATIO NAL J OURNAL OF EPIDEMIOLOGY
by guest on June 7, 2013http://ije.oxfordjournals.org/Downloaded from
Statistical analysis
We used Cox proportional hazards regression to estimate the
relative risk of the primary and secondary outcomes for
vaccinated cohort members compared with unvaccinated cohort
members during time periods defined by influenza surveil-
lance.
30
In analyses of each of the hospitalization outcomes, we
allowed for recurrent events by using a multiple failure time-
proportional hazards model based on counting processes.
31
To account for changing vaccination status during the period of
vaccine availability, we incorporated a time-varying vaccination
status variable into the Cox models.
32,33
At the September 1 start
of each study year, all cohort members were defined as
unvaccinated. Persons who were vaccinated during the study
year then changed to vaccinated status on the day following
vaccination and retained that status through the August 31 end
of the study year. The estimated relative risk was therefore based
on the number of events, the number at risk, and the vaccination
status of each study participant on the exact day at which each
event occurred during the study period, and thus accounted for
differences in cohort vaccination coverage over time. In addition,
models included an interaction term between vaccination status
and each time period to allow the association between influenza
vaccination and risk of the outcome to vary in periods before,
during, and after influenza season.
The primary models were adjusted for sex and age (5 year age-
groups through age >85 years). For comparison, secondary
models were also adjusted for disease covariates. We fit models
that combined data across all study years, and we also examined
single year models to assess variability by study year. In the
analyses across all study years, age and disease covariates were
time-varying variables updated annually on the September 1
start date of each study year. Analyses were conducted using Proc
PHREG from SAS Version 8.2 (Cary, NC), using the Breslow
method for tied follow-up times.
Results
The study cohort included 72 527 seniors, who contributed a
total of 338 264 person-years of observation during the 8 year
study period. During each year, ~44 000 seniors were evaluated,
and influenza vaccine coverage ranged from 68 to 74% during
the study years (Table 1). Across the study period, persons
who had been assigned diagnosis codes indicative of chronic
conditions, with the exception of dementia, contributed a
greater proportion of vaccinated than unvaccinated person-time
(Table 2).
In analyses across all years, the relative risks of the primary
outcomes of death and PI hospitalization were lowest in the
Table 1 Study year characteristics
Study year (September through August)
1995 1996 1997 1998 1999 2000 2001 2002
Influenza season defined by national surveillance
Onset November 13 November 18 December 8 December 14 October 18 November 13 November 19 December 2
Peak December 4 December 2 January 5 January 25 December 13 January 1 January 28 January 13
End April 15 April 7 March 23 April 12 March 13 April 2 May 20 April 28
Influenza season defined by local surveillance
Onset October 28 November 9 December 13 December 27 November 28 December 17 December 16 February 2
Peak November 1 November 16 January 3 January 17 November 28 January 14 January 13 March 2
End January 20 February 8 February 28 April 4 January 23 March 18 March 10 April 6
Number of cohort
members evaluated
42 152 43 039 45 200 45 651 44 416 44 806 45 443 46 767
Total person-years
assessed
40 319 40 974 42 802 42 572 42 036 42 524 42 842 44 192
Number of deaths 836 813 814 854 843 815 867 911
Number of pneumonia
or influenza
hospitalizations
571 597 637 643 695 599 644 680
Date of first influenza
vaccine administration
in the study cohort
September 5 September 16 September 4 September 1 September 15 September 21 September 7 September 27
Date by which at least x% of influenza vaccinations given to study cohort members during the study year had been administered
50% October 20 October 23 October 19 October 22 October 16 November 21 November 10 November 12
75% October 26 October 29 October 25 November 3 October 26 December 5 November 15 November 15
90% November 7 November 7 November 5 November 11 November 4 December 11 November 27 November 21
Vaccination coverage
in the study
cohort, as of
December 31 (%)
72 73 73 71 74 68 70 69
x% represents 50, 75, and 90%.
EVIDENCE OF BIAS IN ESTIMAT ES OF INFLUENZA VACCINE EFF ECTIVENESS 339
by guest on June 7, 2013http://ije.oxfordjournals.org/Downloaded from
period before influenza season (Figure 1), and increased
progressively in the influenza and post-influenza periods.
These results, which were based on time intervals defined by
national influenza surveillance data, did not differ substantively
from the results of analyses based on time intervals defined by
local influenza surveillance data. For example, in the analyses
based on intervals defined by local surveillance, the relative risk
of death was 0.41 before influenza season, 0.54 during influenza
season, and 0.74 after influenza season, compared with estimates
of 0.39, 0.56, and 0.74, respectively, for analyses based on
national surveillance.
In analyses of the secondary hospitalization outcomes, a similar
temporal trend was found, with the lowest point estimates of the
relative risk in the before influenza period (Table 3). Of the
secondary hospitalization outcomes, the lowest estimates of the
relative risk for vaccinated persons compared with unvaccinated
persons were reported for the outcome of injury or trauma
hospitalization, which was a control outcome selected because it
should be unrelated to influenza infection.
Adjustment for the disease covariates defined by diagnosis
codes consistently resulted in lower estimates of the relative risk
compared with the estimates derived from the primary age- and
sex-adjusted models, across all outcomes and in all time periods
(Table 3).
Estimates of the relative risk of death and PI hospitalization in
the before influenza period were robust to alteration of the date
of onset of that period. In analyses of the interval defined as
starting on the date by which 50% of cohort vaccinations had
been distributed and ending at the onset of influenza season,
the relative risk of death was 0.38 and of PI hospitalization was
0.70 (Table 4).
Within the influenza season, the point estimate of the relative
risk of death for vaccinated persons compared with unvaccinated
persons was lowest in the early influenza period (0.46) and then
increased progressively through the peak (0.50) and late (0.69)
influenza periods. For the outcome of PI hospitalization, the
relative risk estimates varied somewhat between the early (0.82),
peak (0.74), and late (0.89) influenza periods, but the confidence
Table 2 Study population characteristics
Characteristic
Vaccinated
person-time,
%(n 5205 472
person-years)
Unvaccinated
person-time,
%(n 5 132 792
person-years)
Age group (yr)
65–74 50.0 53.9
75–84 40.9 36.0
>85 9.2 10.1
Male 42.7 41.9
Conditions defined by diagnosis codes assigned during the
baseline period
Hypertension 26.9 23.9
Lung disease 25.4 21.5
Heart disease 23.8 20.2
Diabetes 13.5 11.8
Cancer 11.6 9.7
Lipid disorders 8.7 7.1
Atrial fibrillation 5.6 4.6
Dementia 3.2 3.9
Renal disease 2.3 2.1
Vasculitis or rheumatologic
disease
2.2 1.8
Indicators of medical utilization during the baseline period
Pneumonia hospitalization 0.7 0.6
>12 outpatient visits 33.3 26.5
Pneumonia or influenza hospitali zation
All cause mortality
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
Relative Risk in vaccinated vs unvacc inated
Before influenza season Influenz a season After influenza season
Time Period
Figure 1 Relative risk (and 95% CI) of all cause mortality and pneumonia or influenza hospitalization in vaccinated seniors compared with
unvaccinated seniors, during periods before, during, and after influenza seasons, September 1995 through August 2003.
340 INTERNATIO NAL J OURNAL OF EPIDEMIOLOGY
by guest on June 7, 2013http://ije.oxfordjournals.org/Downloaded from
intervals overlapped, and all of the within-influenza season point
estimates were higher than the before influenza season point
estimate of 0.72.
The results of the analyses of individual study years, though
more variable, were consistent with the results of the analyses
across the entire study period (Table 5). In every study year,
except 2002, the relative risk of all cause mortality was lowest in
the period before influenza season and increased progressively
in the periods during and after influenza season. Due to the
lower number of events, estimates of the relative risk of PI
hospitalization and injury or trauma hospitalization were
more variable, and in general the confidence intervals of the
before influenza and during influenza estimates overlapped
substantially.
Table 3 Relative risk of death and hospitalization for vaccinated seniors compared with unvaccinated seniors in intervals before, during, and after
influenza season, across all study years
Primary model, adjusted
for age and sex
Secondary model, adjusted for age,
sex, and disease covariates
b
Outcome Time period
a
Relative risk of
outcome in vaccinated
vs unvaccinated 95% CI
Relative risk of
outcome in vaccinated
vs unvaccinated 95% CI
All cause mortality
(n 5 6753)
Before influenza 0.39 0.33–0.47 0.36 0.30–0.44
During influenza 0.56 0.52–0.61 0.51 0.47–0.55
After influenza 0.74 0.67–0.80 0.66 0.61–0.72
Pneumonia or influenza
hospitalization (n 5 5066)
Before influenza 0.72 0.59–0.89 0.65 0.53–0.80
During influenza 0.82 0.75–0.89 0.71 0.65–0.78
After influenza 0.95 0.85–1.07 0.82 0.73–0.92
Ischaemic heart disease
hospitalization (n 5 20 658)
Before influenza 1.06 0.96–1.18 0.92 0.83–1.02
During influenza 1.13 1.08–1.91 0.95 0.90–0.99
After influenza 1.23 1.16–1.29 1.02 0.96–1.08
Congestive heart failure
hospitalization (n 5 10 607)
Before influenza 0.94 0.81–1.08 0.80 0.70–0.93
During influenza 1.00 0.94–1.07 0.82 0.77–0.88
After influenza 1.07 0.99–1.16 0.87 0.81–0.94
Cerebrovascular disease
hospitalization (n 5 5219)
Before influenza 0.85 0.69–1.05 0.81 0.66–0.99
During influenza 0.93 0.85–1.03 0.87 0.79–0.96
After influenza 0.89 0.81–0.99 0.83 0.75–0.92
Injury or trauma
hospitalization (n 5 5319)
Before influenza 0.67 0.55–0.82 0.66 0.54–0.80
During influenza 0.88 0.79–0.96 0.85 0.77–0.94
After influenza 0.85 0.77–0.94 0.83 0.75–0.91
a
For each of the eight study years, the period before influenza season was defined as September 1 to the onset of influenza season (the first week with at least
50 influen za isolates reported); influenza season was defined as the onset through the end of influenza season (the last week with at least 50 influenza isolates
reported); and the period after influenza season was defined as the week following the end of influenza season through August 31. These time periods were based
on national influenza viral surveillance reports.
b
In addition to age and sex, the secondary models also included covariates for atrial fibrillation, heart disease, lung disease, diabetes mellitus, dementia, renal
disease, cancer, vasculitis, and rheumatologic disease, hypertension, lipid disorders, pneumonia hospitaliza tion in previous year, and 12 or more outpatient visits
in previous year defined by methods reported in previous HMO-based studies of influenza vaccine effectiveness.
Table 4 Relative risk of death, and pneumonia or influenza hospitalization for vaccinated seniors compared with unvaccinated seniors in intervals
within the before, during, and after influenza periods
Time period Definition
RR of all cause
mortality in vaccinated
vs unvaccinated (95% CI)
RR of pneumonia or influenza
hospitalization in vaccinated
vs unvaccinated (95%CI)
‘High vaccination’
before influenza
Interval from the date by which 50% of influenza
vaccinations had been distributed to the onset
of influenza season
0.38 (0.31–0.45) 0.70 (0.56–0.88)
Early influenza Onset of influenza season to the peak influenza period 0.46 (0.39–0.53) 0.82 (0.69–0.97)
Peak influenza 5 weeks spanning the 2 weeks before and after the
week of peak viral circulation
0.50 (0.43–0.58) 0.74 (0.64–0.86)
Late influenza The week after the peak period through the end
of influenza season
0.69 (0.65–0.87) 0.89 (0.77–1.02)
Post-influenza End of influenza season through May 31 0.73 (0.65–0.81) 0.97 (0.82–1.15)
Summer June 1 through August 31 0.73 (0.65–0.81) 0.94 (0.81–1.09)
EVIDENCE OF BIAS IN ESTIMAT ES OF INFLUENZA VACCINE EFF ECTIVENESS 341
by guest on June 7, 2013http://ije.oxfordjournals.org/Downloaded from
Discussion
In this study, the reductions in risk observed in the before
influenza period suggest the presence of bias due to preferential
receipt of vaccine by relatively healthy seniors on the estimates
of influenza vaccine effectiveness observed during influenza
season. Among this large cohort of Group Health seniors,
we found reductions in risk of all cause mortality and of PI
hospitalization in vaccinated persons compared with unvaccin-
ated persons during influenza season that were consistent
with estimates reported by previous observational studies. For
example, our age- and sex-adjusted estimate of the relative risk of
death during influenza season of 0.56 is identical to the corre-
sponding estimate reported for the 1999/2000 influenza season
in a large cohort study of seniors in three HMO populations,
11
and our estimate of the relative risk of PI hospitalization of
0.82 is very similar to the corresponding estimate of 0.80 reported
in that study.
In contrast to previous cohort studies, we also evaluated the
period before influenza season and found the greatest reductions
in risk of death and PI hospitalization during that period. Those
estimates were robust to variation in the definition of the
onset of the pre-influenza period, and the survival analysis
methods we used accounted for changes in the vaccination status
of individuals during each study year. The reductions in risk
observed before influenza season almost certainly could not be
due to a true vaccine effect, and most likely were due to
underlying differences between the vaccinated and unvaccinated
groups. The magnitude of the bias demonstrated by the
associations of vaccination with risk of all cause mortality and
PI hospitalization before influenza season was sufficient to
account entirely for the associations observed during influenza
season. The movement of the measures of the vaccine association
towards the null in later time periods is compatible with a
reduction in the differences of the outcome risk between the
vaccinated and unvaccinated groups over time.
Nearly all of the previous observational studies that addressed
the seasonal specificity of estimates of influenza vaccine
effectiveness did so by comparing differences in risk between
vaccinated and unvaccinated persons during influenza season
with differences in risk during post-influenza periods or during
peri-influenza periods that included post-influenza inter-
vals.
1,6,9–11,24,34
The results of our study suggest that the
observation of a greater reduction in risk during influenza season
compared with a later period could be due to a decrease in the
magnitude of the differences between vaccinated and unvac-
cinated persons over time, and so is not necessarily evidence of
a true vaccine effect during influenza season. Therefore,
evaluation of a before influenza period is needed in order to
appropriately interpret the relative risk estimates observed in
the influenza and post-influenza periods.
Only one prior study reported comparison of an influenza
period with a pre-influenza period. That case-control study, of
influenza vaccination and risk of hospitalization for pneumonia
in the elderly, reported a reduction in the risk of pneumonia
during the influenza season [odds ratio (OR), 0.69; 95% CI, 0.49–
0.96] but not before influenza season (OR, 0.98; 95% CI, 0.69–
1.39).
7
While these findings are consistent with a seasonal
specificity of the vaccine effect for the outcome of hospitalized
pneumonia, the study was subject to some limitations. Case
Table 5 Relative risk (and 95% CI) of death, pneumonia or influenza hospitalization, and injury or trauma hospitalization for vaccinated seniors compared with unvaccinated seniors in periods
before, during, and after influenza season, by study year
Study year
Outcome Time period
a
1995 1996 1997 1998 1999
b
2000
b
2001 2002
All cause
mortality
Before influenza 0.23 (0.12–0.44) 0.58 (0.36–0.93) 0.47 (0.33–0.67) 0.44 (0.32–0.61) 0 0 0.52 (0.26–1.03) 0.20 (0.12–0.35)
During influenza 0.55 (0.45–0.68) 0.62 (0.50–0.77) 0.63 (0.49–0.82) 0.71 (0.56–0.90) 0.39 (0.32–0.49) 0.47 (0.38–0.59) 0.55 (0.45–0.66) 0.70 (0.57–0.87)
After influenza 0.74 (0.58–0.95) 0.62 (0.48–0.79) 0.70 (0.56–0.89) 0.86 (0.66–1.12 0.74 (0.58–0.93) 0.91 (0.71–1.16) 0.86 (0.65–1.15) 0.57 (0.45–0.72)
Pneumonia
or influenza
hospitalization
Before influenza 0.95 (0.51–1.75) 0.59 (0.35–1.02) 0.65 (0.42–0.99) 0.66 (0.43–0.99) 2.85 (1.01–8.00) 1.99 (0.26–15.56) 0.51 (0.24–1.11) 0.88 (0.50–1.54)
During influenza 0.76 (0.59–0.99) 0.65 (0.52–0.83) 1.06 (0.79–1.43) 0.94 (0.72–1.21) 0.68 (0.54–0.84) 0.99 (0.77–1.28) 0.87 (0.69–1.09) 0.79 (0.63–1.01)
After influenza 0.96 (0.69–1.33) 1.17 (0.82–1.68) 0.94 (0.70–1.25) 1.05 (0.75–1.46) 0.89 (0.66–1.19) 1.01 (0.75–1.36) 0.68 (0.47–0.97) 0.95 (0.69–1.31)
Injury or
trauma
hospitalization
Before influenza 0.49 (0.29–0.83) 0.92 (0.56–1.52) 0.70 (0.45–1.09) 0.62 (0.41–0.94) 0 3.22 (0.74–14.04) 0.84 (0.42–1.66) 0.61 (0.36–1.03)
During influenza 0.93 (0.72–1.21) 0.70 (0.54–0.92) 0.89 (0.67–1.19) 0.88 (0.65–1.18) 0.94 (0.71–1.24) 0.95 (0.72–1.25) 1.00 (0.78–1.28) 0.76 (0.59–0.96)
After influenza 0.91 (0.69–1.21) 0.80 (0.61–1.06) 0.96 (0.73–1.26) 0.96 (0.71–1.30) 0.86 (0.67–1.13) 0.92 (0.70–1.20) 0.86 (0.62–1.19) 0.65 (0.50–0.83)
Analyses were adjusted for age and sex.
a
For each of the eight September through August study years, the period before influenza season was defined as September 1 to the onset of influenza season (the first week with at least 50 influenza isolates
reported); influenza season was defined as the onset through the end of influenza season (the last week with at least 50 influenza isolates reported); and the period after influenza season was defined as the week
following the end of influenza season through August 31. These time periods were based on national influenza viral surveillance reports.
b
The before influenza period estimates for the 1999 and 2000 study years were unstable due to the small amounts of accrued vaccine-exposed person-time in the before influenza periods in those years. In 1999, the
influenza onset date was very early, and so the before influenza period was short, and in 2000 there was a delay in vaccine availability.
37
As a consequence, in 1999 only 271 vaccine-exposed person-years accrued in
the before influenza period, and in 2000 only 42 vaccine-exposed person-years accrued in that period, compared with an average of 2543 vaccine-exposed person-years in the before influenza periods in other study
years. The calculations of relative risk in the before influenza period in those years were based on zero deaths in vaccine-exposed persons for both years, and on only seven PI hospitalizations in 1999 and one PI
hospitalization in 2000, compared with an average of 32 such PI events in other study years. Thus, it was not possible to obtain more precise estimates of the risk of outcome events in vaccinated persons compared
with unvaccinated persons in the before influenza period in 1999 or 2000.
342 INTERNATIO NAL J OURNAL OF EPIDEMIOLOGY
by guest on June 7, 2013http://ije.oxfordjournals.org/Downloaded from
patients were recruited retrospectively, after the end of
influenza season, and so the survival and health status of
participants at the time of recruitment could have influenced the
likelihood that they would be included in the study population.
Vaccination status was also assessed after the end of influenza
season, and was defined by proxy report for 50% of cases, and so
may have been misclassified. While there is no evidence that the
magnitude of the selection bias or information bias potentially
present in this study differed for subjects with index dates before
or during influenza season, the possibility of differential bias
argues for a cautious interpretation of the time period
comparisons.
In our study, we replicated methods of adjustment for
covariates defined by indicators of medical utilization and by
groupings of diagnosis codes that have been used in previous
studies of HMO populations. We expected that, in analyses of the
before influenza period, when any difference in risk between
vaccinated and unvaccinated persons is presumably due to bias,
proper adjustment would produce relative risk estimates close to
the null value of 1.0. Instead, we found that adjustment for the
covariates we defined led to relative risk estimates for death and
PI hospitalization in the before influenza period that were, if
anything, further from the null than the unadjusted estimates.
This same effect of adjustment that we observed, leading
to lower estimates of the relative risk and therefore greater
estimates of vaccine effectiveness, has been consistently docu-
mented in other vaccine effectiveness studies that adjusted for
health status covariates defined by similar methods.
1,11–14,21,22
For example, in the cohort study of three HMO populations,
adjustment moved the estimate of the relative risk of all cause
mortality in the 1999/2000 influenza season from 0.56 (age- and
sex-adjusted) to 0.50 (age-, sex- and covariate-adjusted), and
moved the estimate of the relative risk of PI hospitalization from
0.80 to 0.71.
11
Failure of this method to adjust for bias may be the
result of the fact that diagnosis codes assigned at medical
encounters are not measures of frailty or disease severity, which
are likely influential factors in the association of influenza
vaccination and the risk of serious health outcomes.
It is important to note that, like other observational studies,
we did not evaluate outcomes specifically due to influenza
infection, because influenza infections are rarely documented by
laboratory testing. The limitation of this approach is that
prevention of influenza-related complications may have relat-
ively little impact on the broader, non-specific study outcomes.
This problem can be illustrated by considering the possible effect
of influenza vaccination on the risk of all cause mortality.
Assuming, for example, that influenza vaccine reduces the risk of
fatal influenza infection by 58%, which is the level of efficacy
against serologically confirmed influenza infection reported by a
randomized trial of older adults,
35
and that influenza infection
accounts for 10% of all deaths during influenza season,
36
then
influenza vaccination would be expected to reduce all cause
mortality during influenza season by 5.8%. The corresponding
estimate of the relative risk of all cause mortality for vaccinated
persons compared with unvaccinated persons, in the absence of
bias, would be ~0.94.
For this reason, our finding that differences in health status
between vaccinated and unvaccinated groups leads to bias in
estimates of influenza vaccine effectiveness against all cause
mortality and other non-specific outcomes does not mean that
there is no effect of vaccination against serious complications of
influenza infection. Our results do suggest, however, that other
methods for evaluations of influenza vaccine effectiveness
should be explored. These methods could include prospective
ascertainment of influenza-specific outcomes, to improve study
sensitivity to detect a true vaccine effect, as well as more accurate
characterization of disease severity and functional status, to
allow better adjustment for confounding. In future studies,
assessment of the effect of adjustment in the before influenza
period may assist in evaluating the degree to which influential
differences between vaccinated and unvaccinated persons are
controlled for in analyses of events during influenza season.
KEY MESSAGESKEY MESSAGESKEY MESSAGESKEY MESSAGESKEY MESSAGESKEY MESSAGESKEY MESSAGESKEY MESSAGESKEY MESSAGESKEY MESSAGES
Numerous observational studies have reported that seniors who receive influenza vaccine are at substantially
lower risk of death and hospitalization during influenza season than unvaccinated seniors, but these estimates
could be influenced by differences in underlying health status between the vaccinated and unvaccinated groups.
Since a protective effect of vaccination should be specific to influenza season, evaluation of non-influenza periods
could indicate the possible contribution of bias to the estimates observed during influenza season.
In a cohort study of 72 527 persons >65 years of age followed during an 8 year period, we evaluated the association
of influenza vaccination and risk of death, and the association of influenza vaccination and risk of pneumonia
hospitalization, in periods before, during, and after influenza season.
We found the greatest reductions in the risk of death and of pneumonia hospitalization in the period before
influenza season, when there should be no true vaccine effect.
The reductions in risk before influenza season suggest the presence of bias due to preferential receipt of
vaccine by relatively healthy seniors on the estimates of influenza vaccine effectiveness observed during influenza
season.
EVIDENCE OF BIAS IN ESTIMAT ES OF INFLUENZA VACCINE EFF ECTIVENESS 343
by guest on June 7, 2013http://ije.oxfordjournals.org/Downloaded from
References
1
Voordouw AC, Sturkenboom MC, Dieleman JP et al. Annual
revaccination against influenza and mortality risk in community-
dwelling elderly persons. JAMA 2004;292:2089–95.
2
Puig-Barbera J, Diez-Domingo J, Perez Hoyos S, Belenguer Varea A,
Gonzalez Vidal D. Effectiveness of the MF59-adjuvanted influenza
vaccine in preventing emergency admissio ns for pneumonia in the
elderly over 64 years of age. Vaccine 2004;23:283–89.
3
Puig-Barbera J, Marquez-Calderon S, Masoliver-Fores A et al. Reduc-
tion in hospital admissions for pneumonia in non-institutionalised
elderly people as a result of influenza vaccination: a case-control study
in Spain. J Epidemiol Community Health 1997;51:526–30.
4
Crocetti E, Arniani S, Bordoni F, Maciocco G, Zappa M, Buiatti E.
Effectiveness of influenza vaccination in the elderly in a community in
Italy. Eur J Epidemiol 2001;17:163–68.
5
Mullooly JP, Bennett MD, Hornbrook MC et al. Influenza vaccination
programs for elderly persons: cost-effectiveness in a health mainten-
ance organization. Ann Intern Med 1994;121:947–52.
6
Foster DA, Talsma A, Furumoto-Dawson A et al. Influenza vaccine
effectiveness in preventing hospitalization for pneumonia in the
elderly. Am J Epidemiol 1992;136:296–307.
7
Ohmit SE, Monto AS. Influenza vaccine effectiveness in preventing
hospitalization among the elderly during influenza type A and type B
seasons. Int J Epidemiol 1995;24:1240–48.
8
Wang ST, Lee LT, Chen LS, Chen TH. Economic evaluation of
vaccination against influenza in the elderly: an experience from a
population-based influenza vaccination program in Taiwan. Vaccine
2005;23:1973–80.
9
Mangtani P, Cumberland P, Hodgson CR, Roberts JA, Cutts FT, Hall AJ.
A cohort study of the effectiveness of influenza vaccine in older people,
performed using the United Kingdom general practice research
database. J Infect Dis 2004;190:1–10.
10
Nichol KL, Baken L, Nelson A. Relation between influenza vaccination
and outpatient visits, hospitalization, and mortality in elderly persons
with chronic lung disease. Ann Intern Med 1999;130:397–403.
11
Nichol KL, Nordin J, Mullooly J, Lask R, Fillbrandt K, Iwane M.
Influenza vaccination and reduction in hospitalizations for cardiac
disease and stroke among the elderly. N Engl J Med 2003;348:1322–32.
12
Nichol KL, Margolis KL, Wuorenma J, Von Sternberg T. The efficacy
and cost effectiveness of vaccination against influenza among elderly
persons living in the community. N Engl J Med 1994;331:778–84.
13
Hak E, Buskens E, van Essen GA et al. Clinical effectiveness of influenza
vaccination in persons younger than 65 years with high-risk medical
conditions: the PRISMA study. Arch Intern Med 2005;165:274–80.
14
Nordin J, Mullooly J, Poblete S et al. Influenza vaccine effectiveness in
preventing hospitalizations and deaths in persons 65 years or older in
Minnesota, New York, and Oregon: data from 3 health plans. J Infect Dis
2001;184:665–70.
15
Shapiro Y, Shemer J, Heymann A et al. Influenza vaccination:
reduction in hospitalizations and death rates among members of
‘‘Maccabi Healthcare Services’’ during the 2000–2001 influenza
season. Isr Med Assoc J 2003;5:706–08.
16
Hak E, Wei F, Grobbee DE, Nichol KL. A nested case–control study of
influenza vaccination was a cost-effective alternative to a full cohort
analysis. J Clin Epidemiol 2004;57:875–80.
17
Landi F, Onder G, Cesari M et al. Effects of influenza vaccination on
mortality among frail, community-living elderly patients: an obser-
vational study. Aging Clin Exp Res 2003;15:254–58.
18
Heymann AD, Shapiro Y, Chodick G et al. Reduced hospitalizations and
death associated with influenza vaccination among patients with and
without diabetes. Diabetes Care 2004;27:2581–84.
19
Fedson DS, Wajda A, Nicol JP, Hammond GW, Kaiser DL, Roos LL.
Clinical effectiveness of influenza vaccination in Manitoba. JAMA
1993;270:1956–61.
20
Nichol KL, Wuorenma J, von Sternberg T. Benefits of influenza
vaccination for low-, intermediate-, and high-risk senior citizens.
Arch Intern Med 1998;158:1769–76.
21
Fleming DM, Watson JM, Nicholas S, Smith GE, Swan AV. Study of the
effectiveness of influenza vaccination in the elderly in the epidemic
of 1989–90 using a general practice database. Epidemiol Infect
1995;115:581–89.
22
Voordouw BC, van der Linden PD, Simonian S, van der Lei J,
Sturkenboom MC, Stricker BH. Influenza vaccination in community-
dwelling elderly: impact on mortality and influenza-associated
morbidity. Arch Intern Med 2003;163:1089–94.
23
Schade CP, McCombs MA. Influenza immunization and mortality
among diabetic Medicare beneficiaries in West Virginia. WVMedJ
2000;96:444–48.
24
Davis JW, Lee E, Taira DA, Chung RS. Influenza vaccination,
hospitalizations, and costs among members of a Medicare managed
care plan. Med Care 2001;39: 1273–80.
25
Centers for Disease Control and Prevention (CDC). Update: influenza
activity—United States and worldwide, 2000–01 season, and com-
position of the 2001–02 influenza vaccine. MMWR Morb Mortal Wkly
Rep 2001;50:466–79.
26
Brammer TL, Murray EL, Fukuda K, Hall HE, Klimov A, Cox NJ.
Surveillance for influenza—United States, 1997–98, 1998–99, and
1999–00 seasons. MMWR Surveill Summ 2002;51:1–10.
27
Brammer TL, Izurieta HS, Fukuda K et al. Surveillance for influenza—
United States, 1994–95, 1995–96, and 1996–97 seasons. MMWR
Surveill Summ 2000;49:13–28.
28
Centers for Disease Control and Prevention (CDC). Update: influenza
activity—United States and worldwide, 2002–03 season, and com-
position of the 2003–04 influenza vaccine. MMWR Morb Mortal Wkly
Rep 2003;52:516–21.
29
Centers for Disease Control and Prevention (CDC). Update: Influenza
activity—United States and worldwide, 2001–02 season, and com-
position of the 2002–03 influenza vaccine. MMWR Morb Mortal Wkly
Rep 2002;51:503–06.
30
Cox DR. Regression models and life tables. J R Stat Soc B 1972;34:
187–220.
31
Therneau TM, Grambsch P. Modeling Survival Data: Extending the Cox
Model. New York: Springer-Verlag, 2000.
32
Fisher LD, Lin DY. Time-dependent covariates in the Cox proportional-
hazards regression model. Annu Rev Public Health 1999;20:145–57.
33
Allison PD. Survival Analysis Using the SAS System: A Practical Guide.
Cary, NC: SAS Institute Inc, 2000.
34
Armstrong BG, Mangtani P, Fletcher A et al. Effect of influenza
vaccination on excess deaths occurring during periods of high
circulation of influenza: cohort study in elderly people. BMJ
2004;329:660.
35
Govaert TM, Thijs CT, Masurel N, Sprenger MJ, Dinant GJ,
Knottnerus JA. The efficacy of influenza vaccination in elderly
individuals. A randomized double-blind placebo-controlled trial. JAMA
1994; 272:1661–65.
36
Simonsen L, Reichert TA, Viboud C, Blackwelder WC, Taylor RJ,
Miller MA. Impact of influenza vaccination on seasonal mortality in
the US elderly population. Arch Intern Med 2005;165:265–72.
37
Centers for Disease Control and Prevention (CDC). Updated recom-
mendations from the advisory committee on immunization practices
in response to delays in supply of influenza vaccine for the 2000–01
season. MMWR Morb Mortal Wkly Rep 2000;49:888–92.
344 INTERNATIO NAL J OURNAL OF EPIDEMIOLOGY
by guest on June 7, 2013http://ije.oxfordjournals.org/Downloaded from
