Incorporating model quality information in climate change detection and attribution studies

Abstract

In a recent multimodel detection and attribution (D&A) study using the pooled results from 22 different climate models, the
simulated “fingerprint” pattern of anthropogenically caused changes in water vapor was identifiable with high statistical
confidence in satellite data. Each model received equal weight in the D&A analysis, despite large differences in the skill
with which they simulate key aspects of observed climate. Here, we examine whether water vapor D&A results are sensitive to
model quality. The “top 10” and “bottom 10” models are selected with three different sets of skill measures and two different
ranking approaches. The entire D&A analysis is then repeated with each of these different sets of more or less skillful models.
Our performance metrics include the ability to simulate the mean state, the annual cycle, and the variability associated with
El Niño. We find that estimates of an anthropogenic water vapor fingerprint are insensitive to current model uncertainties,
and are governed by basic physical processes that are well-represented in climate models. Because the fingerprint is both
robust to current model uncertainties and dissimilar to the dominant noise patterns, our ability to identify an anthropogenic
influence on observed multidecadal changes in water vapor is not affected by “screening” based on model quality.

Full text

Page 1

Incorporating model quality information in climate
change detection and attribution studies
B. D. Santera,1, K. E. Taylora, P. J. Glecklera, C. Bonfilsa, T. P. Barnettb, D. W. Pierceb, T. M. L. Wigleyc, C. Mearsd,
F. J. Wentzd, W. Bru ¨ggemanne, N. P. Gillettf, S. A. Kleina, S. Solomong, P. A. Stotth, and M. F. Wehneri
aProgram for Climate Model Diagnosis and Intercomparison, Lawrence Livermore National Laboratory, Livermore, CA 94550;bScripps Institution of
Oceanography, La Jolla, CA 92037;cNational Center for Atmospheric Research, Boulder, CO 80307;dRemote Sensing Systems, Santa Rosa, CA 95401;
eInstitut fu ¨r Unternehmensforschung, Universita ¨t Hamburg, 20146 Hamburg, Germany;fCanadian Centre for Climate Modelling and Analysis,
University of Victoria, Victoria, BC, Canada V8W 3V6;gChemical Sciences Division, National Oceanic and Atmospheric Administration Earth System
Research Laboratory, Boulder, CO 80305;hHadley Centre, U.K. Meteorological Office, Exeter EX1 3PB, United Kingdom; andiLawrence Berkeley
National Laboratory, Berkeley, CA 94720
Edited by Michael E. Mann, Pennsylvania State University, University Park, PA, and accepted by the Editorial Board July 1, 2009 (received for review
February 23, 2009)
In a recent multimodel detection and attribution (D&A) study using
the pooled results from 22 different climate models, the simulated
‘‘fingerprint’’ pattern of anthropogenically caused changes in water
vapor was identifiable with high statistical confidence in satellite
data. Each model received equal weight in the D&A analysis, despite
large differences in the skill with which they simulate key aspects of
observed climate. Here, we examine whether water vapor D&A
results are sensitive to model quality. The ‘‘top 10’’ and ‘‘bottom 10’’
models are selected with three different sets of skill measures and
two different ranking approaches. The entire D&A analysis is then
repeated with each of these different sets of more or less skillful
models. Our performance metrics include the ability to simulate the
mean state, the annual cycle, and the variability associated with El
Nin ˜o. We find that estimates of an anthropogenic water vapor
fingerprint are insensitive to current model uncertainties, and are
governed by basic physical processes that are well-represented in
climate models. Because the fingerprint is both robust to current
model uncertainties and dissimilar to the dominant noise patterns,
our ability to identify an anthropogenic influence on observed mul-
tidecadalchangesinwatervaporisnotaffectedby‘‘screening’’based
on model quality.
climate modeling ? multimodel database ? water vapor
S
complexcausesofrecentclimatechange(1–3).Fingerprintingrelies
on numerical models of the climate system to provide estimates of
both the searched-for fingerprint—the pattern of climate response
to a change in one or more forcing mechanisms—and the back-
ground‘‘noise’’ofnaturalinternalclimatevariability.Todate,most
formaldetectionandattribution(D&A)workhasusedinformation
from only one or two individual models to estimate both the
fingerprint and noise (4–6). Relatively few D&A studies have used
climate data from three or more models (7–13).
The availability of large, multimodel archives of climate model
output has had important implications for D&A research. A
prominent example of such an archive is the CMIP-3 (Coupled
Model Intercomparison Project) database, which was a key
resource for the Fourth Assessment Report of the Intergovern-
mental Panel on Climate Change (IPCC) (14). The CMIP-3
archive enables D&A practitioners to use information from two
dozen of the world’s major climate models and to examine the
robustness of D&A results to current uncertainties in model-
based estimates of climate-change signals and natural variability
noise (10–13).
Multimodel databases offer both scientific opportunities and
challenges. One challenge is to determine whether the informa-
tion from each individual model in the database is equally
reliable, and should be given equal ‘‘weight’’ in a multimodel
D&A study, or in estimating some ‘‘model average’’ projection
ince the mid-1990s, pattern-based ‘‘fingerprint’’ studies have
been the primary and most rigorous tool for disentangling the
of future climate change (15). Previous multimodel D&A inves-
tigations with atmospheric water vapor (10) and sea-surface
temperatures (SSTs) in hurricane formation regions (13)
adopted a ‘‘one model, one vote’’ approach, with no attempt
made to weight or screen models based on their performance in
simulating aspects of observed climate. An important and hith-
erto unexplored question, therefore, is whether the findings of
such multimodel D&A studies are sensitive to model weighting
or screening decisions.
To address this question, objective measures of model perfor-
mance are required. An obvious difficulty is that model errors are
highly complex; they depend on the variable considered, the space
and timescale of interest, the statistical metric used to compare
modeled and observed climatic fields, the exact property of the
fields that is being considered (e.g., mean state, diurnal or annual
cycle, amplitude and structure of variability, and evolution of
patterns),anduncertaintiesintheobservationsthemselves(16–22).
Recent assessments of the overall performance of CMIP-3 models
have relied on a variety of statistical metrics and were primarily
focused on how well these models reproduce the observed clima-
tological mean state (23, 24).*
Here, we revisit our multimodel D&A study with atmospheric
water vapor over oceans (10). We calculate a number of different
‘‘model quality’’ metrics and demonstrate that use of this informa-
tion to screen models does not affect our ability to identify an
externally forced fingerprint in satellite data.
Observational and Model Water Vapor Data
We rely on observational water vapor data from the satellite-
based Special Sensor Microwave Imager (SSM/I). The SSM/I
atmospheric moisture retrievals commenced in late 1987 and are
based on measurements of microwave emissions from the 22-
GHz water vapor absorption line (25–27). Retrievals are un-
available over the highly emissive land surface and sea-ice
Author contributions: B.D.S., K.E.T., P.J.G., C.B., T.P.B., D.W.P., T.M.L.W., C.M., F.J.W., and
W.B.designedresearch;B.D.S.,P.J.G.,andC.B.performedresearch;C.M.,F.J.W.,P.A.S.,and
M.F.W.contributednewreagents/analytictools;B.D.S.,K.E.T.,P.J.G.,C.B.,T.M.L.W.,N.P.G.,
S.A.K., and S.S. analyzed data; and B.D.S., K.E.T., P.J.G., C.B., T.M.L.W., C.M., W.B., N.P.G.,
S.A.K., S.S., P.A.S., and M.F.W. wrote the paper.
The authors declare no conflict of interest.
ThisarticleisaPNASDirectSubmission.M.E.M.isaguesteditorinvitedbytheEditorialBoard.
Freely available online through the PNAS open access option.
1To whom correspondence should be addressed. Email: santer1@llnl.gov.
*The processes affecting the gradual response of the climate system to long-term anthro-
pogenicforcingneednotbethesameasthosecontrollingshorter-timescalephenomena.
For example, model inadequacies in simulating the diurnal cycle do not necessarily
translate to a deficient simulation of long-term responses.
This article contains supporting information online at www.pnas.org/cgi/content/full/
0901736106/DCSupplemental.
14778–14783 ?
PNAS ?
September 1, 2009 ?
vol. 106 ?
no. 35www.pnas.org?cgi?doi?10.1073?pnas.0901736106

Page 2

regions. Our focus is therefore on W, the total column water
vapor over oceans for a near-global domain.†
As noted above, fingerprint studies require estimates of both the
climate-change signal in response to external forcing and the noise
of internal climate variability. We obtain signal estimates from
simulations with historical changes in natural and anthropogenic
forcings (‘‘20CEN’’ runs) and noise information from control
integrations with no forcing changes.‡We use 20CEN and control
integrations from 22 different climate models in the CMIP-3
archive. These are the same models that were used in our original
water vapor D&A study (10).
Strategy for Assessment of Model Quality
Fig. 1 illustrates why it may be useful to include model quality
information in multimodel D&A studies. The figure shows the
simulated and observed temporal standard deviation of ?W?, the
spatial average of atmospheric water vapor over near-global
oceans.§Results are given for monthly and interannual-timescale
fluctuations in ?W?. On both timescales, the simulated variability in
20CEN runs ranges from one-third to two-and-a-half times the
amplitude of the observed variability.
Are such variability differences between models and observa-
tions of practical importance in multimodel D&A studies? Most
D&A studies routinely apply some form of statistical test to check
the consistency between observed residual variability (after re-
moval of an estimated externally forced signal) and model control
run variability (4, 7–13), and many studies compare power spectra
oftheobservedandmodeledvariablesbeinganalyzed(12,13).Our
focus here is not on formal statistical tests or spectral density
comparisons; instead, it is calculating metrics that provide more
direct information regarding the fidelity with which models simu-
late the amplitude and structure of key modes of natural internal
variability.
AlthoughourD&Astudyinvolveswatervaporonly,wecompute
performance metrics for water vapor and SST. We examine SST
data because observed SST datasets are 130–150 yr in length and
therefore provide a better constraint on model-based estimates of
decadalvariabilitythantheshort(21-yr)SSM/Irecord.Information
on low-frequency variability is crucial for D&A applications, be-
cause it constitutes the background noise against which we attempt
to identify a slowly evolving anthropogenic signal. All SST-based
model quality metrics were calculated using observations from the
NOAA Extended Reconstructed SST (ERSST) dataset (28).
WeevaluatemodelperformanceinsimulatingWandSSTinfive
different regions. The first is the 50°N–50°S ocean domain used in
our previous water vapor D&A work. The next three regions were
chosen because they provide information on model errors in
simulating three characteristic modes of natural climate variability:
the El Nin ˜o/Southern Oscillation (ENSO), the Pacific Decadal
Oscillation (PDO), and the Atlantic Multidecadal Oscillation
(AMO).¶The final region comprises tropical oceans (30°N–30°S)
and is of interest because of claims that modeled and observed
atmospheric temperature changes differ significantly in the tropics.
We analyze model performance in simulating the mean state,
annual cycle, and amplitude and structure of variability.?There are
10meanstatediagnostics(twovariables?fiveregions).Eachmean
state metric is simply a measure of the absolute value of the
climatological annual-mean model bias. The 10 annual cycle diag-
nostics involve the correlations between the simulated and ob-
servedclimatologicalmeanannualcyclepatterns.The50variability
metrics** are measures of model skill in simulating the amplitude
and pattern of observed variability on monthly, interannual, and
decadal timescales. The rationale for examining model perfor-
mance on different timescales is that model variability errors are
complex and frequency-dependent (29).
All 70 metrics are normalized by the intermodel standard
deviationofthestatisticalpropertybeingconsidered.Thisallowsus
to combine information from the mean, annual cycle, and variabil-
itymetricsaswellasfromdifferentclimatevariablesandgeograph-
ical regions. Details regarding the definition and calculation of our
modelperformancemetricsaregiveninthesupportinginformation
(SI) Appendix.
Results from Model Quality Assessment
Results for 40 of the 70 individual metrics are shown in Fig. 2. To
illustratethecomplexityofmodelerrors,weusetheexampleofthe
†OurD&Astudyareaencompassesalloceansbetween50°Nand50°S.Thisdomainwaschosen
to minimize the effect of model-versus-SSM/I water vapor differences associated with inac-
curate simulation of the latitudinal extent of ice margins.
‡Theexternalforcingsimposedinthe20CENexperimentsdifferedbetweenmodelinggroups.
The most comprehensive experiments included changes in both natural external forcings
(solar irradiance and volcanic dust loadings in the atmosphere) and in a wide variety of
anthropogenic influences (such as well-mixed greenhouse gases, ozone, sulfate and black
carbonaerosols,andlandsurfaceproperties).Detailsofthemodels,20CENexperiments,and
control integrations are given in the SI Appendix.
§Here and subsequently, ? ? denotes a spatial mean.
¶ENSOvariabilitycanbecharacterizedinanumberofdifferentways.Weanalyzewatervapor
and SST changes over the Nin ˜o 3.4 region (5°N–5°S; 170°W–120°W). The PDO and AMO
regions used here are 20°N–60°N; 115°W–115°E and 20°N–60°N; 75°W–0°, respectively.
?We do not calculate metrics that gauge model performance in simulating observed water
vapor and SST trends. Results could be biased toward identification of an anthropogenic
fingerprintbyfirstselectingasubsetofmodelswithgreaterskillinreplicatingobservedtrends
andthenusingthesamesubsetinaD&Aanalysisthatcomparesmodeledandobservedtrend
behavior.
**For the higher-frequency variability comparisons, there are a total of 40 metrics: two
variables(SSTandW)?fiveregions(oceans50°N–50°S,ENSO,PDO,andAMOregions,and
tropical oceans) ? two statistical attributes (variability amplitude and pattern) ? two
timescales(monthlyandinterannual).Forcomparisonsofdecadalvariability,thereareonly
10 diagnostics, because these are meaningful to compute for SST only (see the text). All
variabilitypatternmetricsarecenteredcorrelations,withremovalofthespatialmeansofthe
two fields being compared.
0.20.40.6 0.81
Monthly variability of water vapor (kg/m2)
0
0.2
0.4
0.6
0.8
Interannual variability of water vapor (kg/m2)
CCSM3
GFDL-CM2.0
GFDL-CM2.1
GISS-EH
GISS-ER
MIROC3.2(medres)
MIROC3.2(hires)
MIUB-ECHO/G
MRI-CGCM2.3.2
PCM
UKMO-HadCM3
UKMO-HadGEM1
BCCR-BCM2.0
CGCM3.1(T47)
CGCM3.1(T63)
CNRM-CM3
CSIRO-Mk3.0
ECHAM5/MPI-OM
FGOALS-g1.0
GISS-AOM
INM-CM3.0
IPSL-CM4
Observations (SSM/I)
Observations
Fig. 1.
atmosphericwatervapor.ObservationsarefromtheSSM/Idataset(25,26);model
dataarefrom71realizationsof20thcenturyclimatechangeperformedwith22
different models (see SI Appendix). All variability calculations rely on monthly
mean values of ?W?, the spatial average of total atmospheric moisture over
near-global oceans. Model and observational ?W? data were first expressed as
anomalies relative to climatological monthly means over the period 1988–1999
and then linearly detrended. We computed temporal standard deviations from
boththeunfilteredandfilteredanomalydata.Thelatterweresmoothedbyusing
a filter with a half-power point at ?2 years. The raw and filtered standard
deviations provide information on monthly and interannual-timescale variabil-
ity, respectively. All calculations were over the 144-month period from January
1988toDecember1999(theperiodofmaximumoverlapbetweentheSSM/Idata
and most 20CEN simulations). The dashed gray lines are centered on the
observations.
Comparison of the simulated and observed temporal variability of
Santer et al.PNAS ?
September 1, 2009 ?
vol. 106 ?
no. 35 ?
14779
ENVIRONMENTAL
SCIENCES

Page 3

UKMO-HadCM3 model (developed at the UK Meteorological
Office Hadley Centre). Consider first the results for the absolute
bias in the climatological mean state (Fig. 2A). HadCM3 has
relativelysmallbiasvaluesforbothwatervaporandSST,exceptfor
SSTs in the PDO region. When models are ranked parametrically
on the basis of the ‘‘average error’’ results in Fig. 2A, HadCM3 has
the lowest bias values and is therefore ranked first.
In terms of its simulation of the climatological annual cycle
pattern(Fig.2B)andtheamplitudeofmonthlyvariability(Fig.2C),
HadCM3 also performs well relative to its peers and is ranked
seventh and fifth, respectively. For the monthly variability pattern,
however, HadCM3 has a large error for water vapor in the PDO
region (Fig. 2D). This one component has a marked influence on
HadCM3’s low overall ranking (18th) for the monthly variability
pattern. For interannual and decadal variability (not shown),
HadCM3 ranks 10th and first in terms of its variability amplitude
and15thand14thintermsofitsvariabilitypattern.Asisclearfrom
the HadCM3 example and the other model results in Fig. 2,
assessments of the relative skill of the CMIP-3 models are sensitive
to a variety of analyst choices.
This message is reinforced in Fig. 3, which shows that for our
selectedvariables,regions,anddiagnostics,therearenostatistically
significant relationships between model skill in simulating the
climatological mean state and model skill in capturing either the
observed annual cycle or the amplitude and pattern of monthly
variability. Similar findings have been obtained in related studies
(19, 20, 22, 23). One possible interpretation of this result is that the
spatial averages of observed climatological annual means provide a
relativelyweakconstraintonoverallmodelperformance.Modeling
groups attempt to reduce biases in these large-scale climatological
averages by adjusting poorly known physical parameters (and by
fluxcorrection,whichstillisusedinseveraloftheCMIP-3models).
Observed annual cycle and variability patterns offer more stringent
tests of model performance. Reliable reproduction of these more
challenging observational targets is difficult to achieve through
tuning alone—accurate representation of the underlying physics is
of greater importance.
0
1
2
3
4
5
Model bias error
Oceans (50oN-50oS)
Nin~o3.4 region
AMO region
PDO region
Tropical oceans
-3
-2
-1
0
1
2
3
4
Model pattern error
Average error
0
1
2
3
4
5
Model amplitude error
Small error
Oceans (50oN-50oS)
Nin~o3.4 region
AMO region
PDO region
Tropical oceans
-2
-1
0
1
2
3
4
5
Model pattern error
Climatological mean
Annual cycle pattern
Amplitude of monthly variability
CCSM3
GFDL-CM2.0
GISS-EH
GFDL-CM2.1
GISS-ER
MIROC3.2(medres)
MIROC3.2(hires)
MIUB-ECHO/G
MRI-CGCM2.3.3
PCM
UKMO-HadCM3
UKMO-HadGEM1
BCCR-BCM2.0
CGCM3.1(T47) CGCM3.1(T63)
CNRM-CM3
CSIRO-Mk3.0
ECHAM5/MPI-OM
FGOALS-g1.0
GISS-AOM
INM-CM3.0
IPSL-CM4
CCSM3
GFDL-CM2.0
GFDL-CM2.1
GISS-EH
GISS-ER
MIROC3.2(medres)
MIROC3.2(hires)
MIUB-ECHO/G
MRI-CGCM2.3.2
PCM
UKMO-HadCM3
UKMO-HadGEM1
BCCR-BCM2.0
CGCM3.1(T47)CGCM3.1(T63)
CNRM-CM3
CSIRO-Mk3.0
ECHAM5/MPI-OM
FGOALS-g1.0
GISS-AOM
INM-CM3.0
IPSL-CM4
Monthly variability pattern
Water vapor
SST
Small error
Small error
Small error
A
B
CD
Fig. 2.
of metrics used in the ranking of
model performance. The statistics are
measuresofhowwell22ofthemodels
in the CMIP-3 database reproduce key
features of observed water vapor and
SST behavior in five different geo-
graphical regions. The metrics shown
here are a subset of the full suite of
metrics that we applied for model
ranking, and are for the mean state
(A), annual cycle pattern (B), ampli-
tude of monthly variability (C), and
pattern of monthly variability (D). For
models with multiple 20CEN realiza-
tions, values of metrics are averaged
over realizations. The black dots la-
beled ‘‘average error’’ represent the
arithmeticaverage(foreachmodel)of
the 10 metric values (2 variables ? 5
regions). (A and C) Small values of the
normalized metrics indicate greater
skill in simulating the mean state and
the amplitude of monthly variability.
(B and D) Negative values of the nor-
malized pattern correlation metrics
denote greater skill in simulating the
annual cycle and monthly variability
patterns.
Results for four different sets
11.5
Mean state
22.5
-0.5
0
0.5
1
1.5
2
Annual cycle pattern
CCSM3
GFDL-CM2.0
GFDL-CM2.1
GISS-EH
GISS-ER
11.5
Mean state
22.5
0
0.5
1
1.5
2
2.5
3
Amplitude of monthly variability
MIROC3.2(medres)
MIROC3.2(hires)
MIUB-ECHO/G
MRI-CGCM2.3.2
PCM
1 1.5
Mean state
22.5
0
0.5
1
1.5
2
2.5
3
Monthly variability pattern
UKMO-HadCM3
UKMO-HadGEM1
BCCR-BCM2.0
CGCM3.1(T47)
CGCM3.1(T63)
CNRM-CM4
CSIRO-Mk3.0
ECHAM5/MPI-OM
FGOALS-g1.0
GISS-AOM
INM-CM3.0
IPSL-CM4
Correlation = -0.053
Correlation = +0.135
Correlation = +0.021
A
B
C
Fig. 3.
in simulating the mean state and skill in
simulating the annual cycle pattern (A),
amplitude of monthly variability (B), and
monthly variability pattern (C). Results
plotted are the ‘‘average errors’’ shown
and described in Fig. 2. The black lines are
the fitted least-squares regression lines.
Models to the left of the vertical dashed
graylinearerankedinthetop10basedon
values of the mean state metric ? ˆ. Models
below the horizontal dashed gray line are
rankedinthetop10basedonvaluesofthe
annual cycle pattern metric ?ˆ(A), the vari-
abilityamplitudemetric?ˆ(B),andthevari-
ability pattern metric ? ˆ (C). The gray
shadedregionindicatestheintersectionof
the two sets of top 10 models plotted in
each graph.
Relationship between model skill
14780 ?
www.pnas.org?cgi?doi?10.1073?pnas.0901736106Santer et al.

Page 4

The final stage in our model quality assessment is to combine
information from different performance metrics, which we accom-
plish in three different ways. The three combinations involve the 10
mean state and 10 annual cycle diagnostics (M?AC), the 25
variability amplitude and 25 variability pattern metrics (VA?VP),
and the 70 mean state, annual cycle, and variability diagnostics
(ALL).Individualvaluesofthesemetricsareaveraged,yieldingthe
ˆQ1,ˆQ2, andˆQ3statistics, which are used for the parametric ranking
of the CMIP-3 models (see SI Appendix). The nonparametric rank
issimplytheaverageoftheindividualranksratherthantheaverage
of individual metric values.
The overall ranking results are shown in Fig. 4. A number of
interesting features are evident. First, only three models (MRI-
CGCM2.3.2, UKMO-HadGEM1, and IPSL-CM4) are consistently
ranked within the top 10 CMIP-3 models based on both ranking
approaches and all three sets of performance criteria (M?AC,
VA?VP, and ALL). None of the top four models determined with
the M?AC metrics (Fig. 4A) is also in the top four based on the
VA?VP metrics (Fig. 4B). These results support our previous
findingthatassessmentsofmodelqualityaresensitivetothechoice
of statistical properties used in model evaluation.
Second, there is also some sensitivity to the choice of ranking
procedure, particularly for the VA?VP and ALL statistics (Fig. 4
B and C). In each of these two cases, the nonparametric and
parametric ranking approaches identify slightly different sets of
‘‘top10’’models.Only8modelsareintheintersectionofthesesets.
Third, higher horizontal resolution does not invariably lead to
improved model performance. The CMIP-3 archive contains two
models (the Canadian Climate Centre’s CGCM3.1 and the Japa-
nese MIROC3.2) that were run in both higher- and lower-
resolution configurations. The lower-resolution version of
CGCM3.1 outperforms the higher-resolution version in terms of
the M?AC diagnostics but not for the VA?VP metrics. The
reverse applies to the MIROC3.2 model. The lack of a consistent
benefit of higher resolution is partly due to our focus on temper-
ature and moisture changes over oceans. The performance im-
provement related to higher resolution is more evident over land
areas with complex topography (30).
Detection and Attribution Analysis
We now apply the same multimodel D&A method used by Santer
et al. (10). Instead of employing all 22 CMIP-3 models in the D&A
analysis, we restrict our attention to 10-member subsets of the 22
models. These subsets are determined by ranking models on the
basis of the three different sets of metrics (M?AC, VA?VP, and
ALL) and two different ranking approaches (parametric and
nonparametric). From each of these six ranking sets, we select the
top 10 and bottom 10 models, yielding 12 groups of 10 models.
Fingerprints are calculated in the following way. For each set of
10 models, we determine the multimodel average of the atmo-
spheric moisture changes over the period 1900–1999.††The finger-
print is simply the first empirical orthogonal function (EOF) of the
multimodel average changes in water vapor.
Because 10 modeling groups used anthropogenic forcings only,
whereas the other 12 applied a combination of anthropogenic and
natural external forcings (see SI Appendix), we expect the multi-
model fingerprint to down-weight the contribution of natural
external forcing to the fingerprint. However, previous work has
found that the fingerprints estimated from combined historical
changes in anthropogenic and natural external forcing are very
similar to those obtained from ‘‘anthropogenic only’’ forcing (10).
We infer from this that anthropogenic forcing is the dominant
influence on the changes in atmospheric moisture over the 20th
century and that the multimodel fingerprint patterns are not
distorted by the absence of solar and volcanic forcing in 10 of the
22 models analyzed here.‡‡
There is pronounced similarity between the fingerprint patterns
estimated from the 12 subsets of CMIP-3 models (Fig. 5). All 12
patterns show spatially coherent water vapor increases, with the
largest increases over the warmest ocean areas. There are no
systematic differences between the fingerprints estimated from
different sets of metrics, different ranking procedures, or from the
top 10 or bottom 10 models. This indicates that the structure of the
water vapor fingerprint is primarily dictated by the zero-order
physics governing the relationship between surface temperature
and column-integrated water vapor (25, 31).
For each of our 12 subsets of CMIP-3 models, estimates of
natural internal variability are obtained by concatenating the 10
individual control runs of that subset, after first removing residual
drift from each control (Fig. S1 in SI Appendix). The leading EOF
patterns estimated from the concatenated control runs are remark-
ably similar. Each displays the horseshoe-shaped pattern charac-
teristic of the effects of ENSO variability on atmospheric moisture
††Thiscalculationinvolvesaveragingtheensemblemeanwatervaporchangesofeachmodel—
i.e., averaging the 20CEN realizations of an individual model before averaging over models
(seeSIAppendix).Notethatuseofwatervapordatafortheentire20thcentury(ratherthan
simplytheperiodofoverlapwithSSM/I)providesalessnoisyestimateofthetruewatervapor
responsetoslowlyvaryingexternalforcingsandaresponsethatismoresimilaracrossmodels.
‡‡Because volcanic effects on climate have pronounced structure in space and time, they can
and have been identified in D&A studies which include information on the spatiotemporal
evolution of signal and noise (12).
2 4 6 8 10 12 14 16 18 20 22
Parametric rank
2
4
6
8
10
12
14
16
18
20
22
Non-parametric rank
2 4 6 8 10 12 14 16 18 20 22
Parametric rank
2
4
6
8
10
12
14
16
18
20
22
Non-parametric rank
2 4 6 8 10 12 14 16 18 20 22
Parametric rank
2
4
6
8
10
12
14
16
18
20
22
Non-parametric rank
Mean state and annual cycle
A
Amplitude and pattern of variability
B
All diagnostics
C
Fig.4.
ofmodelskillinsimulatingtheobservedmeanstateandannualcycle(A),theamplitudeandpatternofvariability(B),andthecombinedmeanstate,annualcycle,and
variabilityproperties(C).TheQˆ1,Qˆ2,andQˆ3statisticsareaveragesofthenormalizedvaluesof20meanstateandannualcyclemetrics(M?AC),50variabilityamplitude
and variability pattern metrics (VA?VP), and 70 combined metrics (ALL). In the nonparametric ranking procedure, models are ranked from 1 to 22 for each of the 70
metrics,andtheindividualranksarethenaveragedineachofthethreegroupsofmetrics(M?AC,VA?VP,andALL).Fulldetailsofthestatisticsandrankingprocedures
are given in the SI Appendix. The gray shaded boxes indicate the intersection of the two sets of top 10 models. See Fig. 3 for key.
Parametricandnonparametricrankingof22CMIP-3models.TheparametricrankingisbasedontheQˆ1,Qˆ2,andQˆ3statistics,whichare(respectively)measures
Santer et al.PNAS ?
September 1, 2009 ?
vol. 106 ?
no. 35 ?
14781
ENVIRONMENTAL
SCIENCES

Page 5

(Fig. S2 in SI Appendix). Unlike the fingerprints, the leading noise
modes have both positive and negative changes in water vapor.
ThesimilarityofthenoisemodesinFig.S2occursdespitethefact
thatindividualmodelscanhavenoticeabledifferencesinthespatial
structure of their leading mode of water vapor variability (Fig. S3
in SI Appendix). One possible explanation for this result is that
errors in the pattern of the dominant noise mode in individual
models are quasirandom; these random error components are
reduced when the leading noise mode is estimated from a suffi-
ciently large number of concatenated model control runs (22).
The final step was to repeat the multimodel D&A analysis of
Santer et al. (10) with updated SSM/I observations, 12 different
fingerprints(Fig.5),and12model-basednoiseestimates(Fig.S2in
SI Appendix). The D&A analysis was performed 144 times, using
each possible combination of fingerprint and noise (Fig. 6). We do
not employ any form of fingerprint optimization to enhance signal-
to-noise (S/N) ratios (3–7, 10). Our D&A method is fully described
in the SI Appendix.
In each of the 144 cases, the model-predicted fingerprint in
response to external forcing can be positively identified in the
observed water vapor data (Fig. 6). S/N ratios for signals calculated
over the 21-year period 1988–2008 are always above the nominal
5%significancethresholdandexceedthe1%thresholdin62of144
cases. These results illustrate that our ability to identify externally
forced changes in water vapor is not affected by the ‘‘model
screening’’ choices we have made.
Note that there are systematic differences between S/N ratios
estimated with top 10 and bottom 10 models, with ratios for the
latter larger in all 6 cases (Fig. 6). This result occurs because many
of the models ranked in the bottom 10 underestimate the observed
variability of water vapor, thereby spuriously inflating S/N ratios
(Fig. S4 in SI Appendix). In models with more realistic represen-
tations of the mean state, annual cycle, and variability, S/N ratios
are smaller but consistently above the stipulated 5% significance
threshold.
Conclusions
We have shown that the positive identification of an externally
forced fingerprint in satellite estimates of atmospheric water vapor
changes is robust to current model uncertainties. Our ability to
identify this fingerprint is not affected by restricting our original
multimodel D&A study (10) to smaller subsets of models with
superior performance in simulating certain aspects of observed
water vapor and SST behavior. In fact, we find that even models
with noticeable errors in water vapor and SST yield positive
detection of an externally forced fingerprint.
The ubiquitous detection of an externally forced fingerprint
is due to several factors. First, the structure of the water vapor
A M+AC (N-TT; 92.7%)
E
VA+VP (N-TT; 91.4%)
C (N-BT; 88.3%)
A+VP (N-BT; 91.8%)
M+AC (P-TT; 92.7%)
G
VA+VP (P-TT; 94.0%)
M+AC (P-BT; 88.3%)
VA+VP (P-BT; 91.1%)
I ALL (N-TT; 90.0%
M+A
V
)
ALL (N-BT; 90.4%) ALL (P-TT; 91.3%) ALL (P-BT; 91.1%)
EOF LOADING
-3
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
B
C
D
F
H
J
K
L
Fig. 5.
nallyforcedchangesinwatervaporover
near-global oceans. Fingerprints were
estimated from 12 different 10-member
sets of model 20CEN simulations, as de-
scribedinFig.6andtheSIAppendix.The
fingerprintistheleadingEOFofthemul-
timodel average change in water vapor
over the 20th century. The first four fin-
gerprints(A–D)wereestimatedfromthe
top 10 (TT) and bottom 10 (BT) models,
with nonparametric (N) and parametric
(P)rankingsbasedontheM?ACmetrics
(see Fig. 4). The fingerprints in E–H were
estimated from models ranked with the
VA?VP pattern statistics. The final four
fingerprints (I–L) were calculated from
models ranked with a combination of
mean state, annual cycle, and variability
metrics(ALL).Allfingerprintcalculations
wereperformedonacommon10°?10°
latitude/longitude grid. The variance ex-
plained by the leading mode ranges
from 88.3% to 94.0%.
Model fingerprints of exter-
123456789101112
Test number
1
1.5
2
2.5
3
Signal-to-noise ratio
Mean+annual cycleVariabilityAll diagnostics
5%
N
N
P
P
N
N
P
P
N
N
P
P
Light: Test performed with top ten models
Dark: Test performed with bottom ten models
1%
Fig.6.
text, 12 different sets of 10 models were used to calculate 12 externally forced
fingerprintsand12estimatesofnaturalinternalvariability.TheD&Aanalysiswas
then performed 144 times, with all possible combinations of fingerprint and
noise.In‘‘Test1,’’forexample,theD&Aanalysiswasrun12times,eachtimewith
thesameconcatenatedcontrolruns(fromthetop10modelsdeterminedwiththe
M?ACmetricsandnonparametricranking),butwithadifferentfingerprint(see
Fig. 5). The height of each colored bar is the average of the 12 S/N values for the
current test. The black error bars denote the maximum and minimum S/N ratios
and are a measure of the effects of fingerprint uncertainty on S/N. The signal in
the S/N ratio is the linear trend over the period 1988–2008 in Z(t), the projection
of the SSM/I water vapor data onto the fingerprint estimated from the current
10-membersetof20CENruns.Thenoiseisthestandarddeviationofthesampling
distribution of 21-year trends. This distribution is estimated by fitting nonover-
lapping21-yeartrendstoN(t),thetimeseriesoftheprojectionofthecurrentset
of10concatenatedcontrolrunsontothefingerprint.Detectionoftheexternally
forced fingerprint in observed data occurs when the S/N ratio exceeds and
remains above a stipulated 5% significance threshold. The 1% significance
threshold also is shown.
SensitivityofS/Nratiostomodelqualityinformation.Asdescribedinthe
14782 ?
www.pnas.org?cgi?doi?10.1073?pnas.0901736106Santer et al.