Jeffrey S. Whitaker’s research while affiliated with National Oceanic and Atmospheric Administration and other places

The intersection between classical data assimilation methods and novel machine learning techniques has attracted significant interest in recent years. Here we explore another promising solution in which diffusion models are used to formulate a robust nonlinear ensemble filter for sequential data assimilation. Unlike standard machine learning methods, the proposed Ensemble Score Filter (EnSF) is completely training-free and can efficiently generate a set of analysis ensemble members. In this study, we apply the EnSF to a surface quasi-geostrophic model and compare its performance against the popular Local Ensemble Transform Kalman Filter (LETKF), which makes Gaussian assumptions in the analysis step. Numerical tests demonstrate that EnSF maintains stable performance in the absence of localization and for a variety of experimental settings. We find that while LETKF maintains optimal performance in the case of linear observations of the entire state and a perfect model, EnSF shows improvements over LETKF when nonlinear observations are assimilated and the system is subject to unexpected model errors. A spectral decomposition of the analysis results in this nonlinear observation regime shows that the largest improvements over LETKF occur at large scales (small wavenumbers) where LETKF lacks sufficient ensemble spread. Overall, this initial application of EnSF to a geophysical model of intermediate complexity motivates further developments of the algorithm for more realistic problems.

There has been a recent surge in development of accurate machine learning (ML) weather prediction models, but evaluation of these models has mainly been focused on medium‐range forecasts, not their performance in cycling data assimilation (DA) systems. Cycling DA provides a statistically optimal estimate of the system state, which can then be used as initial conditions for model prediction, given observations and previous model forecasts. Here, real surface pressure observations are assimilated into several popular ML models using an ensemble Kalman filter, where accurate ensemble covariance estimation is essential to constrain unobserved state variables from sparse observations. In this cycling DA system, deterministic ML models accumulate small‐scale noise until they diverge. Mitigating this noise with a spectral filter can stabilize the system, but with larger errors than traditional models. Perturbation experiments illustrate that these models do not accurately represent short‐term error growth, leading to poor estimation of cross‐variable covariances.

There has been a recent surge in development of accurate machine learning (ML) weather prediction models, but evaluation of these models has mainly been focused on medium-range forecasts, not their performance in cycling data assimilation (DA) systems. Cycling DA provides a statistically optimal estimate of model initial conditions, given observations and previous model forecasts. Here, real surface pressure observations are assimilated into several popular ML models using an ensemble Kalman filter, where accurate ensemble covariance estimation is essential to constrain unobserved state variables from sparse observations. In this cycling DA system, deterministic ML models accumulate small-scale noise until they diverge. Mitigating this noise with a spectral filter can stabilize the system, but with larger errors than traditional models. Perturbation experiments illustrate that these models do not accurately represent short-term error growth, leading to poor estimation of cross-variable covariances.

Hyperspectral infrared (IR) satellites can provide high-resolution vertical profiles of the atmospheric state, which significantly contributes to the forecast skill of numerical weather prediction, especially for regions with sparse observations. One challenge in assimilating the hyperspectral radiances is how to effectively extract the observation information, due to the interchannel correlations and correlated observation errors. An adaptive channel selection method is proposed, which is implemented within the data assimilation scheme and selects the radiance observation with the maximum reduction of variance in observation space. Compared to the commonly used channel selection method based on the maximum entropy reduction (ER), the adaptive method can provide flow-dependent and time-varying channel selections. The performance of the adaptive selection method is evaluated by assimilating only the synthetic Fengyun-4A ( FY-4A ) GIIRS IR radiances in an observing system simulation experiment (OSSE), with model resolutions from 7.5 to 1.5 km and then 300 m. For both clear-sky and all-sky conditions, the adaptive method generally produces smaller RMS errors of state variables than the ER-based method given similar amounts of assimilated radiances, especially with fine model resolutions. Moreover, the adaptive method has minimum RMS errors smaller than or approaching those with all channels assimilated. For the intensity of the tropical cyclone, the adaptive method also produces smaller errors of the minimum dry air mass and maximal wind speed at different levels, compared to the ER-based selection method. Significance Statement Assimilating satellite radiances has been essential for numerical weather prediction. Hyperspectral infrared satellites provide high-resolution vertical profiles for the atmospheric state and can further improve the numerical weather prediction. Due to limited computational resources, and correlated observations and associated errors, efficient and effective ways to assimilate the hyperspectral radiances are needed. An adaptive channel selection method that is incorporated with data assimilation is proposed. The adaptive channel selection can effectively extract the information from hyperspectral radiances under both clear- and all-sky conditions, with increased model resolutions from kilometers to subkilometers.

Climate Diagnostics and Prediction Workshop Digest

The Joint Effort for Data assimilation Integration (JEDI) is an international collaboration aimed at developing an open software ecosystem for model agnostic data assimilation. This paper considers implementation of the model-agnostic family of the local volume solvers in the JEDI framework. The implemented solvers include the Local Ensemble Transform Kalman Filter (LETKF), the Gain form Ensemble Transform Kalman Filter (GETKF), and the optimal interpolation variant of the LETKF filter (LETKF-OI). This paper documents the implementation choices and strategies that allow model agnostic implementation. We also document an expansive set of localization approaches that includes generic distance-based localization, localization based on modulated ensemble products, but also localizations specific to ocean (based on the Rossby radius of deformation), and land (based on the terrain difference between observation and model grid point). Finally, we apply the developed solvers in a limited set of experiments, including single-observation experiments in atmosphere and ocean, and cycling experiments for the ocean, land, and aerosol assimilation. We also provide a proof of concept that illustrates how JEDI Ensemble Kalman Filter solvers can be used in a strongly coupled framework providing increments to the ocean based on the combined observations from the ocean and the atmosphere.

Weather forecasts made with imperfect models contain state‐dependent errors. Data assimilation (DA) partially corrects these errors with new information from observations. As such, the corrections, or “analysis increments,” produced by the DA process embed information about model errors. An attempt is made here to extract that information to improve numerical weather prediction. Neural networks (NNs) are trained to predict corrections to the systematic error in the National Oceanic and Atmospheric Administration's FV3‐GFS model based on a large set of analysis increments. A simple NN focusing on an atmospheric column significantly improves the estimated model error correction relative to a linear baseline. Leveraging large‐scale horizontal flow conditions using a convolutional NN, when compared to the simple column‐oriented NN, does not improve skill in correcting model error. The sensitivity of model error correction to forecast inputs is highly localized by vertical level and by meteorological variable, and the error characteristics vary across vertical levels. Once trained, the NNs are used to apply an online correction to the forecast during model integration. Improvements are evaluated both within a cycled DA system and across a collection of 10‐day forecasts. It is found that applying state‐dependent NN‐predicted corrections to the model forecast improves the overall quality of DA and improves the 10‐day forecast skill at all lead times.

The analyses produced by intermittent data assimilation methods can be dynamically inconsistent and unbalanced. By gradually distributing the analysis increment along model integration, the incremental analysis update (IAU) is effective to combat the inconsistences and imbalances. Different implementations of IAU with time constant or time‐varying increments using different increment frequencies are systematically evaluated for regional simulations, especially for fast‐moving typhoons. Results show that experiments with IAU generally produce smaller forecast errors of temperature, specific humidity, and wind speed than experiment CTRL without initialization. Three‐dimensional IAUs (3DIAUs) with time‐constant increments have smaller errors than four‐dimensional IAUs (4DIAUs) with time‐varying increments interpolated from 3‐hr and hourly increments. Thus, for regional simulations, 3DIAU that imposes stronger filtering has advantages over 4DIAUs with different increment frequencies. For two typhoon cases, experiments with IAU obtain better intensity and structure of vortex than experiment CTRL; thus, the application of IAU can better retain the observation information and build the improved TC structure. But due to the displacement errors in priors and posteriors, the advantage of 4DIAU that considers the propagation of increment is limited compared to 3DIAU. As a trade‐off between the filtering and time‐varying increment, 4DIAU with 3‐hr increment that considers time‐varying increments compared to 3DIAU but imposes stronger filtering than 4DIAU with hourly increment could be preferred for TCs.

Plain Language Summary Accurate forecasts of the earth system rely on accurate estimates of the initial state of the system, which are created by updating a model forecast with the latest observations using a technique called data assimilation. A crucial aspect of the problem is an accurate estimation of the error of the background model forecast being updated. Ensemble‐based estimates of background‐forecast error covariances are often used, but their accuracy is limited by sample size. To mitigate this, the ensemble‐based estimate is often combined with a static, or climatological estimate that is not dependent on the current model state but can be higher dimensional. Rather than blending the two error covariance estimates directly within the update (the hybrid‐covariance approach), the updates computed separately using the static and ensemble‐based covariances can be combined (the hybrid‐gain approach). The simpler and less expensive hybrid‐gain approach is shown here to perform similarly to the hybrid‐covariance approach, aside from the beneficial impact of extra dynamical balance constraints that are available in the hybrid‐covariance approach but have not yet been implemented in the hybrid‐gain system.