No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Development of Machine Learning-Based Energy Management Agent to Control Fine Dust Concentration in Railway Stations
Abstract
This paper introduces a new strategy for managing energy consumption by employing a constrained deep Q-network (DQN) algorithm to regulate Heating, Ventilation, and Air Conditioning (HVAC) systems. The strategy is grounded in the principles of the Markov Decision Process (MDP), which allows for the creation of a transition kernel. By utilizing an artificial neural network (ANN), we can predict levels of PM 2.5 and PM 10 based on the given conditions and actions. This prediction helps in forming the transition kernel within the MDP framework. Using this predictive model as the transition kernel, we optimize the control policy of the energy management agent through the constrained DQN technique. This technique ensures that actions remain within specific constraints. We validate the effectiveness of our approach through numerical experiments using real data collected from Namgwangju Station. These experiments demonstrate that the introduced constraints successfully keep PM 2.5 and PM 10 levels within predefined thresholds at the stations. Furthermore, reducing the threshold results in more significant decreases in PM levels, achieved by allocating additional power to blowers and air conditioners.
ResearchGate has not been able to resolve any citations for this publication.
This study developed a reinforcement learning-based energy management agent that controls the fine dust concentration by controlling facilities such as blowers and air conditioners to efficiently manage the fine dust concentration in the station. To this end, we formulated an optimization problem based on the Markov decision-making process and developed a model for predicting the concentration of fine dust in the station by training an artificial neural network (ANN) based on supervised learning to develop the transfer function. In addition to the prediction model, the optimal policy for controlling the blower and air conditioner according to the current state was obtained based on the ANN to which the Deep Q-Network (DQN) algorithm was applied. In the case study, it is confirmed that the ANN and DQN of the predictive model were trained based on the actual data of Nam-Gwangju Station to converge to the optimal policy. The comparison between the proposed method and conventional method shows that the proposed method can use less power consumption but achieved better performance on reducing fine dust concentration than the conventional method. In addition, by increasing the value of the ratio that represents the compensation due to the fine dust reduction, the learned agent achieved more reduction on the fine dust concentration by increasing the power consumption of the blower and air conditioner.
The air pollution caused by particulate matter (PM) has become a public health issue due to the risks to human life and the environment. The PM concentration in the air causes haze and affects the lungs and the heart, leading to reduced visibility, allergic reactions, pneumonia, asthma, cardiopulmonary diseases, lung cancer, and even death. In this context, the development of systems for monitoring, forecasting, and controlling emissions plays an important role. The literature about forecasting systems based on Artificial Neural Networks (ANNs) ensembles has been highlighted regarding statistical accuracy and efficiency. In this paper, trainable and non-trainable combination methods are used for PM10 and PM2.5 (particles with an aerodynamic diameter less than 10 and 2.5 micrometers, respectively) time series forecasting for eight different locations, in Finland and Brazil, for different periods. Trainable ensembles based on ANNs, linear regression, and Copulas are compared with non-trainable combinations (mean and median), single ANNs, and linear statistical approaches. Different models are considered so far, including Autoregressive model (AR), Autoregressive and Moving Average Model (ARMA), Infinite Impulse Response Filters (IIR), Multilayer Perceptron (MLP), Radial Basis Function Networks (RBF), Extreme Learning Machines (ELM), Echo State Networks (ESN), and Adaptive Network Fuzzy Inference System (ANFIS). The use of ANNs ensembles, mainly combined with MLP, leads to a better one step ahead forecasting performance. The use of robust air pollution forecasting tools is prime to assist governments in managing air pollution issues like hospital collapse during adverse air quality situations. In this sense, our study is indirectly related to the following United Nations sustainable development goals: SDG 3 - good health and well-being and SDG 11 - sustainable cities and communities.
From the past few decades, it has been observed that the urbanization and industrialization are expanding in the developed nations and are confronting the overwhelming air contamination issue. The citizens and governments have experienced and expressed the increasingly concerned regarding the impact of air pollution affecting human health and proposed sustainable development for overriding air pollution issues across the worldwide. The outcome of modern industrialization contains the liquid droplets, solid particles and gas molecules and is spreading in the atmospheric air. The heavy concentration of particulate matter of size PM10 and PM2.5 is seriously caused adverse health effect. Through the determination of particulate matter concentration in atmospheric air for the betterment of human being well in primary importance. In this paper machine learning predictive models for forecasting particulate matter concentration in atmospheric air are investigated on Taiwan Air Quality Monitoring data sets, which were obtained from 2012 to 2017. These models were compared with the existing traditional models and perform better in predictive performance. The performance of these models was evaluated with statistical measures: Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Mean Square Error (MSE), and Coefficient of Determination (R²).
Particulate matter (PM2.5) remains one of the most dominant contributors to air pollution in Delhi and its acute or chronic exposures have exerted serious health implications. Hence, it is necessary to accurately predict the magnitude of PM2.5 concentrations in order to develop emission reduction strategies for air quality management. In regard to this, few machine learning techniques have been applied to predict daily PM2.5 concentrations in Delhi. Two Different models i.e. SVM and ANN, were built on the inputs of various meteorological and pollutant parameters corresponding to 2-year period from 2016-18. Performance evaluation of the models for PM2.5 prediction has been executed and the results have been discussed. The results of this simulation exercise indicate that the ANN shows better prediction accuracy than SVM for PM2.5 prediction.
Both long- and short-term exposure to high concentrations of airborne particulate matter (PM) severely affect human health. Many countries now regulate PM concentrations. Early-warning systems based on PM concentration levels are urgently required to allow countermeasures to reduce harm and loss. Previous studies sought to establish accurate, efficient predictive models. Many machine-learning methods are used for air pollution forecasting. The long short-term memory and gated recurrent unit methods, typical deep-learning methods, reliably predict PM levels with some limitations. In this paper, the authors proposed novel hybrid models to combine the strength of two types of deep learning methods. Moreover, the authors compare hybrid deep-learning methods (convolutional neural network (CNN)—long short-term memory (LSTM) and CNN—gated recurrent unit (GRU)) with several stand-alone methods (LSTM, GRU) in terms of predicting PM concentrations in 39 stations in Seoul. Hourly air pollution data and meteorological data from January 2015 to December 2018 was used for these training models. The results of the experiment confirmed that the proposed prediction model could predict the PM concentrations for the next 7 days. Hybrid models outperformed single models in five areas selected randomly with the lowest root mean square error (RMSE) and mean absolute error (MAE) values for both PM10 and PM2.5. The error rate for PM10 prediction in Gangnam with RMSE is 1.688, and MAE is 1.161. For hybrid models, the CNN–GRU better-predicted PM10 for all stations selected, while the CNN–LSTM model performed better on predicting PM2.5.
Particularly in rural settings, there has been little research regarding the health impacts of fine particulate matter (PM2.5) during the wildfire season smoke exposure period on respiratory diseases, such as influenza, and their associated outbreaks months later. We examined the delayed effects of PM2.5 concentrations for the short-lag (1–4 weeks prior) and the long-lag (during the prior wildfire season months) on the following winter influenza season in Montana, a mountainous state in the western United States. We created gridded maps of surface PM2.5 for the state of Montana from 2009 to 2018 using spatial regression models fit with station observations and Moderate Resolution Imaging Spectroradiometer (MODIS) aerosol optical thickness data. We used a seasonal quasi-Poisson model with generalized estimating equations to estimate weekly, county-specific, influenza counts for Montana, associated with delayed PM2.5 concentration periods (short-lag and long-lag effects), adjusted for temperature and seasonal trend. We did not detect an acute, short-lag PM2.5 effect nor short-lag temperature effect on influenza in Montana. Higher daily average PM2.5 concentrations during the wildfire season was positively associated with increased influenza in the following winter influenza season (expected 16% or 22% increase in influenza rate per 1 μg/m³ increase in average daily summer PM2.5 based on two analyses, p = 0.04 or 0.008). This is one of the first observations of a relationship between PM2.5 during wildfire season and influenza months later.
Environmental health in subway stations, a typical type of urban underground space, is becoming increasingly important. Ventilation is the principal measure for optimizing the complex physical environment in a subway station. This paper narratively reviews the environmental and health effects of subway ventilation and discusses the relevant engineering, environmental, and medical aspects in combination. Ventilation exerts a notable dual effect on environmental health in a subway station. On the one hand, ventilation controls temperature, humidity, and indoor air quality to ensure human comfort and health. On the other hand, ventilation also carries the potential risks of spreading air pollutants or fire smoke through the complex wind environment as well as produces continuous noise. Assessment and management of health risks associated with subway ventilation is essential to attain a healthy subway environment. This, however, requires exposure, threshold data, and thereby necessitates more research into long-term effects, and toxicity as well as epidemiological studies. Additionally, more research is needed to further examine the design and maintenance of ventilation systems. An understanding of the pathogenic mechanisms and aerodynamic characteristics of various pollutants can help formulate ventilation strategies to reduce pollutant concentrations. Moreover, current comprehensive underground space development affords a possibility for creating flexible spaces that optimize ventilation efficiency, acoustic comfort, and space perception.
A deep Q network (DQN) (Mnih et al., 2013) is an extension of Q learning, which is a typical deep reinforcement learning method. In DQN, a Q function expresses all action values under all states, and it is approximated using a convolutional neural network. Using the approximated Q function, an optimal policy can be derived. In DQN, a target network, which calculates a target value and is updated by the Q function at regular intervals, is introduced to stabilize the learning process. A less frequent updates of the target network would result in a more stable learning process. However, because the target value is not propagated unless the target network is updated, DQN usually requires a large number of samples. In this study, we proposed Constrained DQN that uses the difference between the outputs of the Q function and the target network as a constraint on the target value. Constrained DQN updates parameters conservatively when the difference between the outputs of the Q function and the target network is large, and it updates them aggressively when this difference is small. In the proposed method, as learning progresses, the number of times that the constraints are activated decreases. Consequently, the update method gradually approaches conventional Q learning. We found that Constrained DQN converges with a smaller training dataset than in the case of DQN and that it is robust against changes in the update frequency of the target network and settings of a certain parameter of the optimizer. Although Constrained DQN alone does not show better performance in comparison to integrated approaches nor distributed methods, experimental results show that Constrained DQN can be used as an additional components to those methods.
Background
Exposure to ambient airborne particulate matter is a major risk factor for mortality and morbidity, associated with asthma, lung cancer, heart disease, myocardial infarction, and stroke, and more recently type 2 diabetes, dementia and loss of cognitive function. Less is understood about differential effects of particulate matter from different sources. Underground railways are used by millions of people on a daily basis in many cities. Poor air exchange with the outside environment means that underground railways often have an unusually high concentration of airborne particulate matter, while a high degree of railway-associated mechanical activity produces particulate matter which is physicochemically highly distinct from ambient particulate matter. The implications of this for the health of exposed commuters and employees is unclear.
Main body
A literature search found 27 publications directly assessing the potential health effects of underground particulate matter, including in vivo exposure studies, in vitro toxicology studies, and studies of particulate matter which might be similar to that found in underground railways. The methodology, findings, and conclusions of these studies were reviewed in depth, along with further publications directly relevant to the initial search results.
In vitro studies suggest that underground particulate matter may be more toxic than exposure to ambient/urban particulate matter, especially in terms of endpoints related to reactive oxygen species generation and oxidative stress. This appears to be predominantly a result of the metal-rich nature of underground particulate matter, which is suggestive of increased health risks. However, while there are measureable effects on a variety of endpoints following exposure in vivo, there is a lack of evidence for these effects being clinically significant as may be implied by the in vitro evidence.
Conclusion
There is little direct evidence that underground railway particulate matter exposure is more harmful than ambient particulate matter exposure. This may be due to disparities between in vivo exposures and in vitro models, and differences in exposure doses, as well as statistical under powering of in vivo studies of chronic exposure. Future research should focus on outcomes of chronic in vivo exposure, as well as further work to understand mechanisms and potential biomarkers of exposure.
Fine or ultrafine particulate matter (PM) is a mixture of solid and liquid particles, including carbon-centered combustion particles, secondary inorganics, and crustal-derived particles. Recently, ambient particulate matter has been regarded as a serious health problem in the Republic of Korea. This review focuses on the importance of PM-induced effects, and the adverse health effects from experimental and epidemiological studies. There is widespread evidence that short-term increases in particulate matter result in increased mortality and morbidity in adults and children, even at low levels. Considerable consistency between results from experimental and epidemiological studies has been observed for many health effects, including pulmonary diseases and cardiovascular diseases, although the biologic mechanisms of the health effects are unclear. The adverse health effects are reported to be more pronounced in susceptible population groups such as the elderly, children, or people with preexisting cardiovascular and respiratory conditions. In addition to short-term exposures, long-term exposure to particulate matter is also associated with an increase in total mortality, cardiopulmonary mortality, and respiratory morbidity. The present review suggests that despite a considerable amount of research over the last decades, both epidemiological and experimental studies are needed to further clarify the issues on the effects of fine and ultrafine particulate matter.
Particulate matter (PM) is one of the major air pollutants in China. Traffic-related microenvironments are the
typical scenarios exposed to high PM concentrations. This paper investigates the personal exposure to PM during
commuting in Nanjing in four transportation modes, i.e. subway, bicycle, bus and walking. The measurements
were conducted in a heavy-traffic street during rush hours in summer and winter. The result reveals significant
PM concentration differences between various commuting modes. Passengers in subway cabin are exposed to
lowest PM1 (38.3 μg/m3) and PM2.5 (54.4 μg/m3) concentrations, while passengers in subway station are exposed
to highest PM2.5 (90.5 μg/m3). Pedestrians are exposed to highest PM1 (59.5 μg/m3). Both outdoor and
indoor-generated particles contribute a lot to the particles in subway station and 63.4% of the PM2.5 generated in
subway station are between 1 μm and 2.5 μm in size. Most particles in subway cabin are from subway station and
most particles in bus cabin come from the outdoor air, while indoor sources contribute little. Spatial particle
concentration variations were observed in subway station. The particle concentration at a deeper level is usually
higher than the concentration at a shallower level. Substantial particles within 1–2.5 μm in size were observed at
the platform and the portion within 1–2.5 μm decreased at shallower levels. The PM inhalation during subway
trip is lowest while the inhalations during walking and cycling are more than 5 times higher. During a short
distance subway trip, the exposure in subway station contributes most of the total particle inhalation during the
entire subway trip.
The particulate matters (PM10 and PM2.5) inside urban subway stations greatly influence indoor air quality and passenger comfort. This study aims to analyze and interpret the concentrations of PM10 and PM2.5, measured in several subway stations from October 9th to 22nd, 2016 in Beijing, China. The overall methodology was based on the Statistical Package for Social Science (SPSS) software while General linear model (GLM) and correlation analysis were further applied to examine the sensitivities of different variables to the particle concentrations. The data analysis showed the average overall mass ratio of PM concentrations inside subway station is about 68.7%, much lower than outdoor condition (79.6%). In the areas of the station hall and platform, the real-time PM10 and PM2.5 concentrations varied periodically. In working and operation offices, all rooms had much higher PM concentrations than the outdoor environment when its pollution level was level 3, in which the facility room reached the highest level, while the closed meeting room had the lowest. Correlation analysis results indicated that PM10 and PM2.5 concentrations were mutually correlated (average R² = 0.854), and a strong linear correlation (R² = 0.897) of the subway-station PM concentrations to the outdoor PM conditions, regardless of the outdoor atmospheric PM concentrations pollution level was. Nevertheless, the impact of passenger number and temperature & humidity on the station PM concentrations was less, when compared to the outdoor environment. This paper is expected to provide useful information for further research and design of effective prevention measures on PM in local subway stations, towards a more sustainable and healthier built environment in the city underground.
The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
To date, reinforcement learning has mostly been studied solving simple learning tasks. Reinforcement learning methods that have been studied so far typically converge slowly. The purpose of this work is thus two-fold: 1) to investigate the utility of reinforcement learning in solving much more complicated learning tasks than previously studied, and 2) to investigate methods that will speed up reinforcement learning.
This paper compares eight reinforcement learning frameworks: adaptive heuristic critic (AHC) learning due to Sutton, Q-learning due to Watkins, and three extensions to both basic methods for speeding up learning. The three extensions are experience replay, learning action models for planning, and teaching. The frameworks were investigated using connectionism as an approach to generalization. To evaluate the performance of different frameworks, a dynamic environment was used as a testbed. The environment is moderately complex and nondeterministic. This paper describes these frameworks and algorithms in detail and presents empirical evaluation of the frameworks.
Of several responses made to the same situation, those which are accompanied or closely followed by satisfaction to the animal will, other things being equal, be more firmly connected with the situation, so that, when it recurs, they will be more likely to recur; those which are accompanied or closely followed by discomfort to the animal will, other things being equal, have their connections with that situation weakened, so that, when it recurs, they will be less likely to occur. The greater the satisfaction or discomfort, the greater the strengthening or weakening of the bond. (Thorndike, 1911) The idea of learning to make appropriate responses based on reinforcing events has its roots in early psychological theories such as Thorndike's "law of effect" (quoted above). Although several important contributions were made in the 1950s, 1960s and 1970s by illustrious luminaries such as Bellman, Minsky, Klopf and others (Farley and Clark, 1954; Bellman, 1957; Minsky, 1961; Samuel, 1963; Michie and Chambers, 1968; Grossberg, 1975; Klopf, 1982), the last two decades have wit- nessed perhaps the strongest advances in the mathematical foundations of reinforcement learning, in addition to several impressive demonstrations of the performance of reinforcement learning algo- rithms in real world tasks. The introductory book by Sutton and Barto, two of the most influential and recognized leaders in the field, is therefore both timely and welcome. The book is divided into three parts. In the first part, the authors introduce and elaborate on the es- sential characteristics of the reinforcement learning problem, namely, the problem of learning "poli- cies" or mappings from environmental states to actions so as to maximize the amount of "reward"
Effects of fine dust on health
- S Lee
- H Kim