Conference PaperPDF Available

Santek, D., J. García-Pereda, C. Velden, I. Genkova, D. Stettner, S. Wanzong, S. Nebuda, M. Mindock, 2014: A new Atmospheric Motion Vector Intercomparison Study. Proceedings, 12th International Winds Workshop, Copenhagen, Denmark.

Authors:
  • NOAA/NCEP/EMC

Abstract and Figures

This study furthers a line of previously completed research regarding the similarities and differences between the operational Atmospheric Motion Vector (AMV) algorithms of various satellite-derived wind producers. By using a common set of MSG/SEVIRI images and ancillary data, past intercomparison studies assessed how the cloudy AMVs from each unique wind producer compared in terms of coverage, speed, direction, and cloud height (Genkova et al. 2008; Genkova et al. 2010). The current study focuses on including the CMA and NWC SAF AMV algorithms in the intercomparison in order to quantify its performance relative to other AMV algorithms, on updating the results of the previous AMV intercomparison studies due to the changes that have occurred since 2009, and lastly, on performing follow up studies, identified in the previous intercomparison work, to analyze particular issues in pursuance of a more complete understanding of how the different AMV algorithms compare. The study finds a mix of both positive and negative results. The different AMV algorithms successfully determine the horizontal and vertical displacements of the moved features, but not all centers define a consistent AMV speed and direction with these displacements. Using the IR brightness temperature for the height assignment, the distribution of AMV heights is highly variable due to the variability of how this representative Temperature is defined. When additional height assignment techniques are used, all centers except JMA improve the AMV validation statistics. Nevertheless, the improvement is limited for some of the centers because of using the improved height assignment techniques in only a small part of the data. The two centers using CCC height assignment method (Eumetsat and NWC SAF) are in general the ones obtaining the best validation statistics. Considering the AMV coverage, important differences occur between centers even in the case where a similar prescribed configuration is used. This paper is a summary of the full AMV intercomparison Technical Report which can be found at: https://www.researchgate.net/publication/264484893_Santek_D_J_Garcia-Pereda_C_Velden_I_Genkova_S_Wanzong_D_Stettner_M_Mindock_2014_2014_AMV_Intercomparison_Study_Report_Comparison_of_NWC_SAFHRW_AMVs_with_AMVs_from_other_producers
Content may be subject to copyright.
A New Atmospheric Motion Vector Intercomparison Study
David Santek
1
, Javier García-Pereda
2
, Chris Velden
1
, Iliana Genkova
3
,
Dave Stettner
1
, Steve Wanzong
1
, Sharon Nebuda
1
, Max Mindock
1
1
CIMSS/UW-Madison, 1225 W. Dayton St., Madison, Wisconsin, 53706 USA
2
NWCSAF/AEMET, Leonardo Prieto Castro 8, Madrid, 28040 Spain
3
IMSG/NOAA/NCEP, 5830 University Research Court, College Park, Maryland, 20740 USA
Abstract
This study furthers a line of previously completed research regarding the similarities and differences
between the operational Atmospheric Motion Vector (AMV) algorithms of various satellite-derived wind
producers. By using a common set of MSG/SEVIRI images and ancillary data, past intercomparison
studies assessed how the cloudy AMVs from each unique wind producer compared in terms of
coverage, speed, direction, and cloud height (Genkova et al. 2008; Genkova et al. 2010).
The current study focuses on including the CMA and NWC SAF AMV algorithms in the
intercomparison in order to quantify its performance relative to other AMV algorithms, on updating the
results of the previous AMV intercomparison studies due to the changes that have occurred since
2009, and lastly, on performing follow up studies, identified in the previous intercomparison work, to
analyze particular issues in pursuance of a more complete understanding of how the different AMV
algorithms compare.
The study finds a mix of both positive and negative results. The different AMV algorithms successfully
determine the horizontal and vertical displacements of the moved features, but not all centers define a
consistent AMV speed and direction with these displacements. Using the IR brightness temperature
(T
BT
) for the height assignment, the distribution of AMV heights is highly variable due to the variability
of how this representative T
BT
is defined. When additional height assignment techniques are used, all
centers except JMA improve the AMV validation statistics. Nevertheless, the improvement is limited
for some of the centers because of using the improved height assignment techniques in only a small
part of the data. The two centers using CCC height assignment method (EUM and NWC) are in
general the ones obtaining the best validation statistics. Considering the AMV coverage, important
differences occur between centers even in the case where a similar prescribed configuration is used.
This paper is a summary of the full AMV intercomparison Technical Report which can be found at:
www.nwcsaf.org/HD/files/vsadoc/CIMSS_AMV_Comparison_FinalReport_04July2014.pdf.
INTRODUCTION
The first Atmospheric Motion Vector (AMV) algorithm intercomparison study analyzed data from five
centers across the world. Since then, AMV algorithms have changed and the AMV algorithms for CMA
and NWC SAF had yet to be compared to any of the other wind producers. So, this report not only
includes these two new AMV algorithms, but it also updates and expands on the previous studies.
The report analyzes the AMV algorithms for the following wind producers. The three-letter
abbreviations are used throughout the remainder of this report.
BRZ: Brazil Weather Forecast and Climatic Studies Center
CMA: China Meteorological Administration
EUM: EUMETSAT (European Organization for the Exploitation of Meteorological Satellites)
JMA: Japan Meteorological Agency
KMA: Korea Meteorological Administration
NOA: National Oceanic and Atmospheric Administration
NWC: NWC SAF (Satellite Application Facility on Support to Nowcasting & Very Short
Range Forecasting)
Output provided by each of the wind producers is analyzed in four independent experiments, each one
designed to measure differences related to a specific aspect of the AMV algorithms. Portions of the
scripts used were first developed in the previous intercomparison study, which therefore allows for
comparisons to be made between the studies (Genkova et al. 2008, 2010).
The AMV output originated from each algorithm’s analysis of a triplet of infrared (10.8µ) Meteosat–9,
full–disk images from 17 September 2012 at 1200, 1215, 1230 UTC, one of which is shown in Figure
1. Additionally, both 6.2µ, 7.3µ, 12.0µ and 13.4µ images and Meteorological Products Extraction
Facility (MPEF) output products “Scene Type and Quality” and Cloud Analysis” for the same slots
were provided, in case AMV producers wanted to use them for the AMV height assignment procedure
in Experiment 4.
Figure 1: Meteosat-9 10.8 µm from 17 September 2012 at 1215 UTC
ECMWF forecast grids for the 12- and 18-hour forecast from the 17 September 2012 0000 UTC run
were provided as ancillary data. They were reformatted to the Meteosat–9 domain with the following
specifications:
135x135 grid centered at 0°N/0°E
Domain: 67°S to 67°N; 67°E to 67°W
spatial resolution
40 vertical levels
Parameters: pressure, geopotential height, temperature, water vapor mixing ratio, ozone
mixing ratio, wind speed, wind direction, and dew point temperature.
Note: The NOAA AMV processing software requires additional parameters, so additional NWP
background data were used.
Each center’s output for the experiments included data for identical variables (Table 1), with three
exceptions:
BRZ did not report QI with forecast (QIWF)
CMA did not report QI without forecast (QINF)
JMA did not report AMV speed for Experiment 1.
The description and configuration of the individual wind retrieval algorithm is extracted from
information provided by each producer in response to a questionnaire. A summary is found in Section
6: Summary of Wind Retrieval Algorithms of the full report.
In this new Intercomparison study, an additional analysis is performed with the goal to quantify the
differences in terms of statistical significance. This is done by using a paired t-test. Paired t-tests,
unlike the standard (Student’s) t-test, assume two data points are related before it determines if they
are statistically different. In our case, each data point from center X is paired, by having both latitude
and longitude coordinates within a specified distance, with its corresponding data point from center Y.
For each of the comparisons between AMV wind producers, paired t-tests are calculated for several
variables in each experiment (horizontal and vertical displacement, speed, direction, pressure, quality
indicator with and without forecast) in order to compare every combination of centers, with a 95%
confidence setting.
Parameter
Code Description
1 IDN Identification number
2 LAT[DEG] Latitude
3 LON[DEG] Longitude
4 TBOX[PIX] Target box size
5 SBOX[PIX] Search box size
6 SPD[MPS] AMV speed
7 DIR[DEG] AMV direction
8 P[HPA] AMV pressure
9 LOWL Low level correction
10 GSPD[MPS]
Background guess wind speed
11 GDIR[DEG] Background guess wind direction
12 ALB[%] Albedo
13 CORR[%] Correlation
14 TMET Brightness temperature
15 PERR[HPA] AMV pressure error
16 HMET Height assignment method
17 QINF[%] QI without forecast
18 QIF[%] QI with forecast
19 HDISP1 Horizontal pixel displacement for first pair
20 VDISP1 Vertical pixel displacement for first pair
21 HDISP2 Horizontal pixel displacement for second pair
22 VDISP2 Vertical pixel displacement for second pair
Table 1: Reported Variables
EXPERIMENTS
Four experiments were designed to test and compare different aspects of the AMV algorithms: target
selection, tracking, cloud height assignment, and quality control.
Experiment 1
The winds producers extracted AMVs from images with a known displacement (four columns left and
two lines down for the second one with respect to the first one, and the third one with respect to the
second one). This allows this experiment to test the tracking step in each AMV algorithm. Because
each one of the three images is identical, the pattern matching code in each algorithm should work
perfectly. For collocated targets, these ”artificial AMVs” are analyzed to quantify differences in the
tracking process, considering the pixel displacement distribution, speed and direction distribution, and
collocation differences.
There are two main positive results from Experiment 1:
All AMV algorithms detect the shift correctly, generally with no more than a 0.1-pixel difference
(related to the use of subpixel tracking implemented by each center).
When using a distance threshold of 35 km, resulting in 10867 collocated vectors, neither the
horizontal nor vertical displacement differences between any two centers are statistically
significant. However, there are numerous couplings of centers for both speed and direction
that result in statistically different displacements.
Considering this, BRZ and CMA appear to have an AMV speed dependence on distance from the
satellite subpoint, which may have been due to the method used to compute the feature displacement
(BRZ) or to a truncation in the speed (CMA).
Experiment 2
The producers extracted AMVs with their standard algorithm configuration, while only using the
MSG/SEVIRI IR10.8µ images and the ECMWF model data for the height assignment. Each AMV
producer’s standard algorithm is defined by the settings (target scene size, search scene size, etc.) it
typically uses. In addition, a best-fit analysis is utilized in order to further investigate differences in
cloud heights.
By allowing each AMV producer to use their typical settings, this experiment tests the target selection
and tracking steps in each algorithm. The differences have been examined through the parameter
distributions, collocation plots, rawinsonde comparisons, model grid comparisons, and best fit height
comparisons.
Surprisingly, despite each algorithm only using the IR brightness temperature (T
BT
), the bulk
distribution of AMV heights is highly variable, and when paired, each combination of two centers has a
statistically significant difference, with most differences ranging from 30 to 80 hPa from each other; the
largest differences occurring with the different centers compared to EUM (up to 130 hPa). This
indicates the variability is likely due to how the representative T
BT
is determined. Regarding the wind
speed, EUM compared to NWC and JMA result in the only two cases not being statistically different.
NWC and EUM is additionally the only combination very close in terms of both speed and direction,
despite having a cloud height bias of 130 hPa. The average speed difference across all centers
ranges from 0.3 to 1.0 ms
-1
.
When the AMVs are compared to rawinsondes (Table 2) or to the NWP background (Table 3), NWC
has the lowest errors (vector RMS of 6 and 5 ms
-1
respectively), while BRZ and EUM have the highest
(vector RMS of 9 ms
-1
). Regarding EUM, the presence of upper-level winds being too low and lower-
level winds being too high, the large differences of heights compared to other centers, and the large
errors compared to rawinsondes suggest the IR brightness temperature height assignment did not
perform well.
Considering the comparison of collocated AMVs (shown in the full report), there are not significant
changes in the validation statistics. NWC, JMA and KMA show the best results while EUM shows
again the worst results.
N Pre Bias Pre RMS Spd Bias Spd RMS
Dir Bias Vec RMS
BRZ 63 0.67 18.81 0.14 5.27 -11.12 9.59
CMA 241 3.60 26,33 0.17 7.51 5.05 8.99
EUM 268 -0.53 26.57 3.09 7.24 0.05 9.43
JMA 177 -2.20 26.26 0.36 6.04 6.07 8.04
KMA 1346 1.19 24.98 -0.02 5.94 9.04 7.91
NOA 361 -1.59 27.14 3.08 6.30 12.84 8.94
NWC 2410 -1.86 26.03 -0.78 4.75 1.53 6.14
Table 2: Experiment 2: AMV comparison (QI without forecast >= 50; for CMA QI with forecast >=50) to rawinsondes
winds within 150 km. N = number of matches; Pre Bias = pressure bias; Pre RMS = pressure RMS; Spd Bias = wind
speed bias; Spd RMS = wind speed RMS; Dir Bias = wind direction bias; Vec RMS = vector RMS. The extreme for each
category is highlighted: Yellow = high value; cyan = low value
N Vec Diff Vec RMS
BRZ 743 7.51 8.89
CMA 3964 7.07 8.22
EUM 5378 6.88 9.73
JMA 3498 4.50 6.05
KMA 26427 5.95 7.88
NOA 8180 6.87 8.79
NWC 43626 4.76 5.68
Table 3: Experiment 2: AMV comparison to NWP background. N = number of matches; Vec Diff = mean vector
difference; Vec RMS = vector RMS. The extreme for each category is highlighted: Yellow = high value; cyan = low value
Experiment 3
The producers extracted AMVs with a prescribed algorithm configuration, while again using the
MSG/SEVIRI IR10.8µ images and the ECMWF model data for the height assignment. The prescribed
algorithm defined several parameters, such as the target scene size and the search scene size.
The prescribed AMV configuration allows for differences in the tracking and quality control steps to be
highlighted, due to the target selection parameters being set constant across AMV producers.
Identically to Experiment 2, the differences are examined through parameter distributions, collocation
plots, rawinsonde comparisons, model grid comparisons, and best fit height comparisons.
The results are very similar to Experiment 2, since the height assignment options are restricted to IR
brightness temperature. There are less collocated vectors (only 370 as opposed to 7050 in
Experiment 2) due to the lower overall number of AMVs when the prescribed target and search box
size are used. Considering this, there are substantially fewer statistical differences between centers in
speed and direction (in fact, differences between all centers for direction are not statistically
significant), although pressure and QI values are still largely statistically different.
Experiment 4
The winds producers extracted AMVs with the same prescribed algorithm configuration as in
Experiment 3, but now using the height assignment method of their choosing (e.g., CO
2
Slicing, H
2
O
Intercept, CCC method with the use of an external cloud product, etc.). A verification to determine if
the new AMV heights improve the statistics over the results from Experiment 3 is also included.
The usage of the prescribed AMV configuration paired with varying height assignment methods
spotlights the differences in the height assignment and quality control steps. The improved height
assignment methods result in a shift in the distribution of AMV pressure, for both upper and lower level
clouds, especially noted for EUM AMVs (whose vector RMS improves from 9-10 to 5-6 ms
-1
), NOA
(from 8 to 7 ms
-1
), and NWC (from 5-6 to 4 ms
-1
), as shown in Table 4 and Table 5.
Other centers (BRZ, CMA, KMA, JMA) have very few AMVs shifted in height, resulting in a smaller
change in the comparison errors against rawinsonde and model grid winds. In any case, the impact of
the additional height assignment methods is positive in all cases except for JMA, for which statistics
degrade in Experiment 4 respect to Experiment 3.
N Pre Bias Pre RMS Spd Bias Spd RMS
Dir Bias Vec RMS
BRZ 153 0.63 9.77 0.55 5.61 -3.07 10.05
CMA 237 -1.11 18.58 -1.30 6.40 5.28 7.74
EUM 307 0.22 22.87 -0.61 4.73 1.99 6.07
JMA 154 -3.00 21.50 -2.26 7.64 8.89 9.60
KMA 326 -0.63 21.91 -0.73 4.72 2.68 6.38
NOA 131 0.35 22.75 1.48 5.79 9.01 7.70
NWC 73 -0.76 17.53 -0.60 3.48 -3.74 4.67
Table 4: Experiment 4: AMV comparison (QI without forecast >= 50; for CMA QI with forecast >=50) to rawinsondes
winds within 150 km. N = number of matches; Pre Bias = pressure bias; Pre RMS = pressure RMS; Spd Bias = wind
speed bias; Spd RMS = wind speed RMS; Dir Bias = wind direction bias; Vec RMS = vector RMS. The extreme for each
category is highlighted: Yellow = high value; cyan = low value
N Vec Diff Vec RMS
BRZ 1590 8.02 9.67
CMA 4743 6.38 7.44
EUM 6583 3.91 5.36
JMA 3514 4.91 6.59
KMA 4574 5.16 6.83
NOA 2274 5.90 7.54
NWC 1419 3.05 4.01
Table 5: Experiment 4: AMV comparison to NWP background. N = number of matches; Vec Diff = mean vector
difference; Vec RMS = vector RMS. The extreme for each category is highlighted: Yellow = high value; cyan = low value
Considering the collocated vectors (numbering 9942), nearly all speed, direction, pressure and QI
differences are significant between all centers. Pressure differences are smaller than in Experiment 3,
although the mean difference for collocated vectors has still a range between 20 and 100 hPa. In the
corresponding statistics against rawinsonde and NWP background winds (shown in the full report),
BRZ has the largest vector RMS values while EUM and NWC show the smallest ones, so showing the
similarities provided by their common height assignment method (CCC method).
The distribution of AMV vectors for Experiment 4 for the different centers, considering high level winds
(above 375 hPa) and low level winds (below 850 hPa) is shown in Figure 2 to Figure 8. The coverage
of EUM AMVs is the most complete; the density of winds in the two mid-latitude cyclones to the
northwest of Africa and in the marine stratus in the Southern Ocean is especially to be noted.
The coverage of BRZ, CMA, NOA and KMA algorithms is good for the high level clouds; however few
low level vectors are detected over the Southern Ocean (in general caused by the AMV level being
defined as higher than 850 hPa). Instead, for the JMA algorithm the coverage of low level winds is
similar to EUM while the coverage at high levels is not as complete.
Finally, the NWC algorithm captures both high and low level winds, although the density of data is
smaller to EUM using the prescribed configuration (equivalent to EUM operational configuration). If the
NWC operational configuration is used (shown in Figure 9), it results in a very high resolution
coverage of AMVs at both high and low levels.
Figures 2 and 3: BRZ and CMA high-level (cyan, above 375 hPa) and low-level (magenta, below 850 hPa) AMVs overlaid
on the Meteosat-9 10.8µ from 17 September 2012 at 1215 UTC
Figures 4 and 5: EUM and JMA high-level (cyan, above 375 hPa) and low-level (magenta, below 850 hPa) AMVs overlaid
on the Meteosat-9 10.8µ from 17 September 2012 at 1215 UTC
Figures 6 and 7: KMA and NOA high-level (cyan, above 375 hPa) and low-level (magenta, below 850 hPa) AMVs
overlaid on the Meteosat-9 10.8µ from 17 September 2012 at 1215 UTC
Figures 8 and 9: NWC high-level (cyan, above 375 hPa) and low-level (magenta, below 850 hPa) AMVs, using the
prescribed and the NWC operational configuration respectively, overlaid on the Meteosat-9 10.8µ from 17 September
2012 at 1215 UTC
CONCLUSIONS
Brazil Weather Forecast and Climatic Studies Center
Results from Experiment 1 indicate an error in determining wind speeds up to 10 ms
-1
, depending on
the distance from the satellite subpoint. However, the best-fit analyses suggest there are still good
AMVs in this dataset, as the best-fit adjustment and the corresponding improvement in statistics
compared to the background is similar to other centers.
China Meteorological Administration
The CMA AMV algorithm performs well in Experiment 1, detecting the correct displacement of the
artificially moved features in all cases, but the AMV comparison to rawinsondes and the NWP
background winds exhibit larger errors than other centers in the other experiments. This may have
been due to a very extensive use of IR-only T
BT
in determining AMV heights.
EUMETSAT
The strengths of the EUM AMV algorithm are highlighted in Experiment 1 and 4; all vector
displacements are correct (Experiment 1) and the statistical comparison of the EUM AMVs to
rawinsondes and the background forecast wind field performs best together with NWC AMVs
(Experiment 4), with a very dense coverage of AMVs at both high and low levels.
In Experiments 2 and 3 on the other hand, the use of IR-only T
BT
results in AMVs being placed several
hundred hPa different than when other techniques could be used (Experiment 4). This conclusion is
confirmed by the high error in the rawinsonde comparison statistics and is likely due to a brightness
temperature that is too warm.
Japan Meteorological Agency
The JMA AMV algorithm performs very well in Experiments 2 and 3, but results from Experiment 4
show a relative degradation of validation statistics when measuring performance against both
rawinsonde and NWP background winds. Specifically, the AMV coverage for JMA is very dense in low
levels, but the upper-level winds are few and do not compare well.
Korea Meteorological Administration
The KMA AMV algorithm performs relatively well in all four Experiments, especially in Experiment 2
and 3. In Experiment 4, AMV coverage is dense for the upper-level winds; however lower-level winds
are assigned too high in the atmosphere.
NOAA
The strength of the NOA algorithm lay in its cloud height determination, as evidenced in Experiment 4;
a substantial number of heights are adjusted (as compared to IR-only T
BT
), which result in an
improvement in the statistical comparison to rawinsondes and the NWP background winds.
NWC SAF
Among all of the centers in the study, the NWC AMV algorithm has in general the best statistics as
compared to rawinsondes and the NWP background winds. Moreover, NWC AMVs with IR-only cloud
height performs better than several other centers using other cloud height techniques.
There are two areas noted for suggestion to improve the NWC AMV algorithm: first, investigate
increasing the coverage of the upper-level winds since it is less dense than several other centers (e.g.,
EUM, NOA, KMA) when the prescribed configuration is used. Second, the IR T
BT
technique used by
NWC may not be the best method for warmer clouds as the lower-level clouds are placed too high in
Experiment 2.
REFERENCES
Genkova, I., R. Borde, J. Schmetz, J. Daniels, C. Velden, K. Holmlund, 2008: Global atmospheric
motion vectors intercomparison study, 9
th
Int. Winds Workshop, Annapolis, MD USA, April 2008.
Genkova, I., R. Borde, J. Schmetz, C. Velden, K. Holmlund, N. Bormann, P. Bauer, 2010: Global
atmospheric motion vector intercomparison study, 10
th
Int. Winds Workshop, Tokyo, Japan, February
2010.
Santek, D., J. García-Pereda, C. Velden, I. Genkova, S. Wanzong, D. Stettner, M. Mindock, 2014:
2014 AMV intercomparison study report. Technical Report available online at:
www.nwcsaf.org/HD/files/vsadoc/CIMSS_AMV_Comparison_FinalReport_04July2014.pdf.
ResearchGate has not been able to resolve any citations for this publication.
Technical Report
Full-text available
Previous Atmospheric Motion Vector (AMV) intercomparison studies, conducted from 2007 to 2009, compared the operational AMV algorithms of various satellite-derived wind producers using a common set of MSG/SEVIRI images and ancillary data. The studies assessed how the cloudy AMVs from the unique wind producers compared in terms of coverage, speed, direction, and cloud height (Genkova et al. 2008; Genkova et al. 2010). The goal of this new study is to: 1: Include the NWC SAF/HRW algorithm in the intercomparison study in order to quantify its performance relative to the other AMV algorithms. 2: Update the results of the previous AMV intercomparison studies because many of the operational AMV algorithms have changed since the last study. 3: Perform follow up studies as identified in the previous intercomparison work, such as considering specific characteristics of the input data and AMV output.
AMV intercomparison study report
  • D Santek
  • J García-Pereda
  • C Velden
  • I Genkova
  • S Wanzong
  • D Stettner
  • M Mindock
Santek, D., J. García-Pereda, C. Velden, I. Genkova, S. Wanzong, D. Stettner, M. Mindock, 2014: 2014 AMV intercomparison study report. Technical Report available online at: www.nwcsaf.org/HD/files/vsadoc/CIMSS_AMV_Comparison_FinalReport_04July2014.pdf.