Project

Sparse Matrix Solvers in Air Quality Models on Parallel Processors

Goal: Replace legacy sparse matrix LU decomposition and solution for aqueous chemistry in environmental and global climate models. See my presentations in conferences at www.cmascenter.org/conferences.cfm

Updates
0 new
4
Recommendations
0 new
0
Followers
0 new
2
Reads
1 new
25

Project log

George Delic
added a research item
This presentation covers thread parallel performance results for CMAQ 5.3 for a 101-day simulation and extends by far the results in [1]. Here attention is focused on the Gear, Rosenbrock, and EBI solvers in the Chemistry Transport Model (CTM), for both FSparse [1], and the legacy JSparse [2] algorithms. The former implements OpenMP thread parallelism for all three solvers in the CTM. The results include execution performance and numerical precision for the first quarter of the 2016 annual CONUS scenario provided by the U.S. EPA [3]. Both the legacy (EPA) JSparse and the FSparse thread parallel versions are compared in a hybrid MPI+OpenMP version on a heterogeneous cluster of 14 nodes with a total of 192 cores. The implementation of thread parallelism in the horizontal advection science procedures (HADV) will also be discussed since these dominate the fraction of total wall clock time with increased number of MPI processes. [1] G. Delic, Modern Environmental Science and Engineering, Vol. 5, Nr.9, 2019, pp. 775-791. Full text available at: https://www.researchgate.net/publication/338581080_A_Thread_Parallel_Sparse_Matrix_Chemistry_Algorithm_for_the_Community_Multiscale_Air_Quality_Model [2] M. Jacobson and R.P. Turco (1994), Atmos. Environ. 28, 273-284 [3] The author gratefully acknowledges help from Kristen Foley (EPA), Ed Anderson (GDIT), and Elizabeth Adams (UNC) in providing model data and resolving implementation issues.
George Delic
added an update
Published in:
Modern Environmental Science and Engineering (ISSN 2333-2581) September 2019, Volume 5, No. 9, pp. 775-791 Doi: 10.15341/mese(2333-2581)/09.05.2019/001 Academic Star Publishing Company, 2019
 
George Delic
added a research item
This presentation reports on integration of a new Chemistry Transport Model (CTM) sparse matrix algorithm (FSparse) as a replacement of the legacy JSparse algorithm in the U.S. EPA Community Multicscale Air Quality (CMAQ) model. This has been implemented in both Rosenbrock and Gear methods for aqueous chemistry in a hybrid MPI and OpenMP implementation. Both methods are well suited for an OpenMP thread-parallel implementation. For a 24 hour scenario, execution performance results for both MPI and OpenMP thread parallel scaling are presented with the CMAQ5.3b release on a heterogeneous cluster of 10 nodes with a total of 128 cores. The FSPARSE version of CMAQ typically provides significant speedup over the standard EPA release without similar precision in predicted species concentration values.
George Delic
added a research item
This is the latest report on HiPERiSM's hybrid (MPI+OpenMP) version of the U.S. EPA's CMAQ model. It is specific to the Gear Solver algorithm and represents the culmination of a decade of development. Results are from HiPERiSM's 2 Tflops 128 core cluster.
George Delic
added an update
Attached is the latest paper in the series at the CMAS meetings. Since this meeting the algorithm has been applied to each single cell in the grid and compared with the usual method that applies to a block of cells. Results for both host CPU and Intel Phi co-processor will be forth comming at a later date. After 8 years of development with application to both EPA's CMAQ model and NASA's GMI Global Climate model, the work is in the completion phase.
 
George Delic
added an update
The complete reorganization into a modular version has been completed in the case of the Rosenbrock algorithm and is now fully debugged. This (modular) version is an update of the previous version reported at the CMAS meeting (see attached file) where some problems have been corrected and work-arounds for compiler issues implemented. Current work is to include this modular version in the Gear algorithm. So far execution with the original (U.S. EPA) version of CMAQ shows that superior precision achieved in the Gear algorithm at the cost of longer execution times. However, lowering the absolute error tolerance brings down the run time close to that of the Rosenbrock algorithm, without any penalty in precision loss. Hence the conclusion is that the Rosenbrock solver should be replaced by the Gear algorithm (aka "the gold standard") in regular CMAQ applications. Details will follow in the next update.
 
George Delic
added an update
Developing and debugging a modular version of my sparse matrix algorithm suitable for porting to either CPU or Intel Phi processors
 
George Delic
added a project goal
Replace legacy sparse matrix LU decomposition and solution for aqueous chemistry in environmental and global climate models. See my presentations in conferences at www.cmascenter.org/conferences.cfm