Project

LEXIS Large scale Execution for Industry and Society

Goal: The LEXIS (Large-scale EXecution for Industry & Society) project aims to build an advanced, geographically- distributed, HPC infrastructure for Big Data analytics within three targeted pilot test-beds. By proposing innovative technologies and exploiting data available from test-bed partners, LEXIS aims to generate valuable outcomes and improve the efficiency and quality of services provided to different stakeholders involved in the test-beds. The developed services will be made available to external stakeholders, with the aim of stimulating the interest of European industries and creating an ecosystem of organisations that could benefit from the implemented platform. LEXIS will address key strategic objectives that will be transversally pursued all across the work. We aim to demonstrate the benefits of our project in the context of three industrial test-beds, leveraging modern technologies from HPC, Big Data, and Cloud computing. We will refer to Key Performance Indications (KPIs) to demonstrate success against the declared objectives within the industrial test-beds. These KPIs are relevant not only for the objectives themselves, but also for the various stakeholders involved.

Date: 1 January 2019 - 1 July 2021

Updates
0 new
0
Recommendations
0 new
0
Followers
0 new
8
Reads
0 new
43

Project log

Paola Mazzoglio
added a research item
The LEXIS (Large-scale EXecution for Industry & Society) H2020 project is building an advanced engineering platform taking advantage of HPC, Cloud solutions and Big Data, leveraging existing HPC infrastructures. In the framework of the LEXIS project, CIMA Research Foundation is running a three nested domain WRF Model with European coverage and radar data assimilation over Italy. WRF data is then processed by ITHACA Extreme Rainfall Detection System (ERDS), an early warning system developed for the monitoring of heavy rainfall events. The WRF-ERDS workflow has been applied to the heavy rainfall event that affected Southern Italy, in particular Calabria Region, at the end of November 2020. Rainfall depths obtained using global-scale rainfall datasets and WRF data have been compared both with rain gauge data and with the daily bulletins issued by the Italian Civil Protection Department. The data obtained by running the WRF-ERDS workflow shows as an advanced engineering platform based on HPC and cloud solutions can provide more detailed forecasts to an early warning system like ERDS.
Antonio Parodi
added a research item
In this work, we describe the integration of Weather and Research Forecasting (WRF) forecasts produced by CIMA Research Foundation within ITHACA Extreme Rainfall Detection System (ERDS) to increase the forecasting skills of the overall early warning system. The entire workflow is applied to the heavy rainfall event that affected the city of Palermo on 15 July 2020, causing urban flooding due to an exceptional rainfall amount of more than 130 mm recorded in about 2.5 h. This rainfall event was not properly forecasted by meteorological models operational at the time of the event, thus not allowing to issue an adequate alert over that area. The results highlight that the improvement in the quantitative precipitation scenario forecast skills, supported by the adoption of the H2020 LEXIS computing facilities and by the assimilation of in situ observations, allowed the ERDS system to improve the prediction of the peak rainfall depths, thus paving the way to the potential issuing of an alert over the Palermo area.
Paola Mazzoglio
added a research item
In the framework of LEXIS (Large-scale EXecution for Industry & Society) H2020 project, CIMA Research Foundation is running a 3 nested domain WRF (Weather Research and Forecasting) model with European coverage and weather radar data assimilation over Italy. Forecasts up to 48 hours characterized by a 7.5 km resolution are then processed by ITHACA ERDS (Extreme Rainfall Detection System), an early warning system for the heavy rainfall monitoring and forecasting. This type of information is currently managed by ERDS together with two global-scale datasets. The first one is provided by NASA/JAXA GPM (Global Precipitation Measurement) Mission through the IMERG (Integrated Multi-satellitE Retrievals for GPM) Early run data, a near real-time rainfall information with hourly updates, 0.1°spatial resolution and a 4 hours latency. The second one is instead provided by GFS (Global Forecast System) at a 0.25° spatial resolution. The entire WRF-ERDS workflow has been tested and validated on the heavy rainfall event that affected the Sardinia region between 27 and 29 November 2020. This convective event significantly impacted the southern and eastern areas of the island, with a daily rainfall depth of 500.6 mm recorded at Oliena and 328.6 mm recorded at Bitti. During the 28th, the town of Bitti (Nuoro province) was hit by a severe flood event. Near real-time information provided by GPM data allowed us to issue alerts starting from the late morning of the 28th. The first alert over Sardinia based on GFS data was provided in the late afternoon of the 27th, about 40 km far from Bitti. In the early morning of the 28th, a new and more precise alert was issued over Bitti. The first alert based on WRF data was instead provided in the morning of the 27th and the system continued to issue alerts until the evening of the 29th, confirming that, for this type of event, precise forecasts are needed to provide timely alerts. Obtained results show how, taking advantage of HPC resources to perform finer weather forecast experiments, it is possible to significantly improve the capabilities of early warning systems. By using WRF data, ERDS was able to provide heavy rainfall alerts one day before than with the other data. The integration within the LEXIS platform will help with the automatization by data-aware orchestration of our workflow together with easy control of data and workflow steps through a user-friendly web interface.
Paola Mazzoglio
added a research item
LEXIS (Large-scale EXecution for Industry and Society) H2020 project is currently developing an advanced system for Big Data analysis that takes advantage of interacting large-scale geographically-distributed HPC infrastructure and cloud services. More specifically, LEXIS Weather and Climate Large-Scale Pilot workflows ingest data coming from different sources, like global/regional weather models, conventional and unconventional meteorological observations, application models and socio-economic impact models, in order to provide enhanced meteorological information at the European scale. In the framework of LEXIS Weather and Climate Large-scale Pilot, CIMA Research Foundation is running a 7.5 km resolution WRF (Weather Research and Forecasting) model with European coverage, radar assimilation over the Italian area, and daily updates with 48 hours forecast. WRF data is then processed by ITHACA ERDS (Extreme Rainfall Detection System - http://erds.ithacaweb.org), an early warning system for the monitoring and forecasting of heavy rainfall events. The WRF model provides more detailed information compared to GFS (Global Forecast Systems) data, the most widely used source of rainfall forecasts, implemented in ERDS also. The entire WRF - ERDS workflow was applied to two of the most severe heavy rainfall events that affected Italy in 2020. The first case study is related to an intense rainfall event that affected Toscana during the afternoon and the evening of 4th June 2020. In this case, the Italian Civil Protection issued an orange alert for thunderstorms, on a scale from yellow (low) to orange (medium) to red (high). In several locations of the northern part of the Region more than 100 mm of rainfall were recorded in 3 hours, corresponding to an estimated return period equal to or greater than 200 years. As far as the 24-hours time interval concerns, instead, the estimated return period decreases to 10-50 years. Despite the slight underestimation, WRF model was able to properly forecast the spatial distribution of the rainfall pattern. In addition, thanks to WRF data, precise information about the locations that would be affected by the event were available in the early morning, several hours before the event affected these areas. The second case study is instead related to the heavy rainfall event that affected Palermo (Southern Italy) during the afternoon of 15th July 2020. According to SIAS (Servizio Informativo Agrometeorologico Siciliano) more than 130 mm of rain fell in about 2.5 hours, producing widespread damages due to urban flooding phenomena. The event was not properly forecasted by meteorological models operational at the time of the event, and the Italian Civil Protection did not issue an alert on that area (including Palermo). During that day, in fact, only a yellow alert for thunderstorms was issued on northern-central and western Sicily. Within LEXIS, no alert was issued using GFS data due to the severe underestimation of the amount of forecasted rainfall. Conversely, a WRF modelling experiment (three nested domain with 22.5, 7.5 and 2.5 km grid spacing, innermost over Italy) was executed, by assimilating the National radar reflectivity mosaic and in situ weather stations from the Italian Civil Protection Department, and it resulted in the prediction of a peak rainfall depth of about 35 mm in 1 hour and 55 mm in 3 hours, roughly 30 km far apart the actual affected area, thus values supportive at least a yellow alert over the Palermo area. Obtained results highlight how improved rainfall forecast, made available thanks to the use of HPC resources, significantly increases the capabilities of an operational early warning system in the extreme rainfall detection. Global-scale low-resolution rainfall forecasts like GFS one are in fact widely known as good sources of information for the identification of large-scale precipitation patterns but lack precision for local-scale applications.
Olivier Terzo
added a research item
The LEXIS Weather and Climate Large-scale Pilot will deliver a system for prediction of water-food-energy nexus phenomena and their associated socio-economic impacts. The system will be based on multiple model layers chained together, namely global weather and climate models, high-resolution regional weather models, domain-specific application models (such as hydrological, forest fire risk forecasts), impact models providing information for key decision and policy makers (such as air quality, agriculture crop production, and extreme rainfall detection for flood mapping). This paper will report about the first results of this pilot in terms of serving model output data and products with Cloud and High Performance Data Analytics (HPDA) environments, on top a Weather Climate Data APIs (ECMWF), as well as the porting of models on the LEXIS Infrastructure via different virtualization strategies (virtual machine and containers).
Olivier Terzo
added a research item
Accurate and rapid earthquake loss assessments and tsunami early warnings are critical in modern society to allow for appropriate and timely emergency response decisions. In the LEXIS project, we seek to enhance the workflow of rapid loss assessments and emergency decision support systems by leveraging an orchestrated heterogeneous environment combining high-performance computing resources and Cloud infrastructure. The workflow consists of three main applications: Firstly, after an earthquake occurs, its shaking distribution (ShakeMap) is computed based on the OpenQuake code. Secondly, if a tsunami may have been triggered by the earthquake, tsunami simulations (first a fast and coarse and later a high-resolution and computationally intensive computation) are performed based on the TsunAWI simulation code that allows for an early warning in potentially affected areas. Finally, based on the previous results, a loss assessment based on a dynamic exposure model using open data such as OpenStreetMap is computed. To consolidate the workflow and ensure respect of the time constraints, we are developing an extension of a time-constrained dataflow model of computation, layered above and below the workflow management tools of both the high-performance computing resources and the Cloud infrastructure. This model of computation is also used to express tasks in the workflow at the right granularity to benefit from the data management optimisation facilities of the LEXIS project. This paper describes the workflow, the computations associated and the model of computation within the LEXIS platform.
Alberto Scionti
added 2 research items
High Performance Computing (HPC) infrastructures (also referred to as supercomputing infrastructures) are at the basis of modern scientific discoveries, and allow engineers to greatly optimize their designs. The large amount of data (Big-Data) to be treated during simulations is pushing HPC managers to introduce more heterogeneity in their architectures, ranging from different processor families to specialized hardware devices (e.g., GPU computing, many-cores, FPGAs). Furthermore, there is also a growing demand for providing access to supercom-puting resources as in common public Clouds. All these three elements (i.e. challenges emerging in scientific and technical domains. The LEXIS project aims to design and set up an innovative computing architecture, where HPC, Cloud and Big-Data solutions are closely integrated to respond to the demands of performance, flexibility and scalability. To this end, the LEXIS architecture leverages on three main distinctive elements: i) resources of supercomputing centers (geographically located in Europe) which are seamlessly managed in a fed-erated fashion; ii) an integrated data storage subsystem, which supports Big-Data ingestion and processing; and iii) a web portal to enable users to easily get access to computing resources and manage their workloads. In addition, the LEXIS architecture will make use of innovative hardware solutions, such as burst buffers and FPGA accelerators, as well as a flexible orchestration software. To demonstrate the capabilities of the devised converged architecture, LEXIS will assess its performance, scalability and flexibility in different contexts. To this end, three computational highly demanding pilot test-beds have been selected as representative of application domains that will take advantage of the advanced LEXIS architecture: i) Aeronautics Computational Fluid Dynamics simulations of complex turbo-machinery and gearbox systems; ii) Earthquake and Tsunami-acceleration of tsunami simulations to enable highly-accurate real-time analysis; and ; and iii) Weather and Climate-enabling complex workflows which combine various numerical forecasting models, from global & regional weather forecasts to specific socioeconomic impact models affecting emergency management (fire & flood), sustainable agriculture and energy production.
High Performance Computing (HPC) infrastructures (also referred to as supercomputing infrastructures) are at the basis of modern scientific discoveries, and allow engineers to greatly optimize their designs. The large amount of data (Big-Data) to be treated during simulations is pushing HPC managers to introduce more heterogeneity in their architectures, ranging from different processor families to specialized hardware devices (e.g., GPU computing, many-cores, FPGAs). Furthermore, there is also a growing demand for providing access to supercomputing resources as in common public Clouds. All these three elements (i.e., HPC resources, Big-Data, Cloud) make “converged” approaches mandatory to addresschallenges emerging in scientific and technical domains. The LEXIS project aims to design and set up an innovative computing architecture, where HPC, Cloud and Big-Data solutions are closely integrated to respond to the demands of performance, flexibility and scalability. To this end, the LEXIS architecture leverages on three main distinctive elements: i) resources of supercomputing centers (geographically located in Europe) which are seamlessly managed in a federated fashion; ii) an integrated data storage subsystem, which supports Big-Data ingestion and processing; and iii) a web portal to enable users to easily get access to computing resources and manage their workloads. In addition, the LEXIS architecture will make use of innovative hardware solutions, such as burst buffers and FPGA accelerators, as well as a flexible orchestration software. To demonstrate the capabilities of the devised converged architecture, LEXIS will assess its performance, scalability and flexibility in different contexts. To this end, three computational highly demanding pilot test-beds have been selected as representative of application domains that will take advantage of the advanced LEXIS architecture: i) Aeronautics – Computational Fluid Dynamics simulations of complex turbo-machinery and gearbox systems; ii) Earthquake and Tsunami – acceleration of tsunami simulations to enable highly-accurate real-time analysis; and ; and iii) Weather and Climate – enabling complex workflows which combine various numerical forecasting models, from global & regional weather forecasts to specific socio-economic impact models affecting emergency management (fire & flood), sustainable agriculture and energy production.
Olivier Terzo
added a project goal
The LEXIS (Large-scale EXecution for Industry & Society) project aims to build an advanced, geographically- distributed, HPC infrastructure for Big Data analytics within three targeted pilot test-beds. By proposing innovative technologies and exploiting data available from test-bed partners, LEXIS aims to generate valuable outcomes and improve the efficiency and quality of services provided to different stakeholders involved in the test-beds. The developed services will be made available to external stakeholders, with the aim of stimulating the interest of European industries and creating an ecosystem of organisations that could benefit from the implemented platform. LEXIS will address key strategic objectives that will be transversally pursued all across the work. We aim to demonstrate the benefits of our project in the context of three industrial test-beds, leveraging modern technologies from HPC, Big Data, and Cloud computing. We will refer to Key Performance Indications (KPIs) to demonstrate success against the declared objectives within the industrial test-beds. These KPIs are relevant not only for the objectives themselves, but also for the various stakeholders involved.