Fig 5 - uploaded by Chanaka Keerthisinghe
Content may be subject to copyright.
Illustration of the modified Markov decision process, which separates the state variables into post decision states and pre-decision states.

Illustration of the modified Markov decision process, which separates the state variables into post decision states and pre-decision states.

Source publication
Article
Full-text available
This paper presents a computationally efficient smart home energy management system (SHEMS) using an approximate dynamic programming (ADP) approach with temporal difference learning for scheduling distributed energy resources. This approach improves the performance of a SHEMS by incorporating stochastic energy consumption and PV generation models o...

Context in source publication

Context 1
... are used in line 12 of Algorithm 1. An example of the new modified MDP is illustrated in Fig. 5, which uses the mean and variation of the stochastic variables to obtain the post-decision and next pre-decision states, respectively. In more detail, at s 1 , there are three possible decisions that takes us to three post-decision states, which correspond to high, middle and lowest states. However, the next pre decision state s 2 ...

Similar publications

Article
Full-text available
This paper presents an integrative demand response (DR) mechanism for energy management of appliances, an energy storage system and an electric vehicle (EV) within a home. The paper considers vehicle-to-home (V2H) and vehicle-to-grid (V2G) functions for energy management of EVs and the degradation cost of the EV battery caused by the V2H/V2G operat...

Citations

... Double-pass ADP algorithms have also been proposed for microgrid and storage dispatch. For example, a computationally efficient smart home energy management system is proposed in [25], considering stochastic energy consumption and photovoltaics (PV) generation models over a horizon of several days. In [26], a double-pass ADP based on a structured lookup table is proposed for optimal scheduling of wind paired with storage and benchmarked against the optimal solution on a library of deterministic and stochastic problems. ...
... Update VFAV n t−1 (S x,n t−1 ) using (9) 22: end for 23: n ← n + 1 24: end while 25 At each iteration n, a sample path ω n is first generated to represent a realization of exogenous information. Forward pass and backward pass are then executed in sequence: ...
Article
Full-text available
Approximate dynamic programming (ADP) is a promising approach for power system scheduling and dispatch under uncertainties. This paper presents an innovative ADP-based dispatch method for a microgrid with intermittent renewable generation, battery energy storage systems, and controllable distributed generators. The proposed ADP algorithm is based on a double-pass value iteration approach and takes advantage of the underlying properties of the microgrid dispatch problem. In the forward pass, decision variables are updated moving forward in time using an ε-greedy strategy to balance exploitation and exploration. In particular, an approximate optimization method is proposed to speed up exploitation. In addition to random exploration, a policy is designed to guide the algorithm to explore some promising solution space in a probabilistic manner. In the backward pass, the value function is updated moving backward in time using the trajectory of states, decisions, and outcomes of the sample path in the forward pass. The proposed method is evaluated through numerical experiments in both deterministic and stochastic environments. Case study results show that the proposed method demonstrates improved performance in both optimization gap and computation time in comparison to conventional methods.
... In this work, a policy is a sequence of on/off status of the HVAC system over a defined time horizon. Let s kþ1 ¼ s M ðs k ; x k Þ describe the evolution from time step k to the next time step, k þ 1, where s M is the underlying mathematical model of the studied system (see Ref. [27] for a detailed HEMS formulation). In this problem, the system model is a system of the ordinary differential equations (ODE) (1) and (2). ...
Article
Full-text available
This work investigates the extent to which phase change material (PCM) in the building's envelope can be used as an alternative to battery storage systems to increase self-consumption of rooftop solar photovoltaic (PV) generation. In particular, we explore the electricity cost-savings and increase in PV self-consumption that can be achieved by using PCMs and the operation of the heating, ventilation, and air conditioning (HVAC) system optimised by a home energy management system (HEMS). In more detail, we consider a HEMS with an HVAC system, rooftop PV, and a PCM layer integrated into the building envelope. The objective of the HEMS optimisation is to minimise electricity costs while maximising PV self-consumption and maintaining the indoor building temperature in a preferred comfort range. Solving this problem is challenging due to PCM's nonlinear characteristics, and using methods that can deal with the resulting non-convexity of the optimisation problem, like dynamic programming is computationally expensive. Therefore, we use multi-timescale approximate dynamic programming (MADP) that we developed in our earlier work to explore a number of Australian PCM scenarios. Specifically, we analyse a large number of residential buildings across five Australian capital cities. We find that using PCM can reduce annual electricity costs by between 10.6% in Brisbane and 19% in Adelaide. However, somewhat surprisingly, using PCM reduces PV self-consumption by between 1.5% in Brisbane and 2.7% in Perth.
... groups points together which are close to each other based on a distance measurement and a minimum number of points [52][53][54][55] Dimensionality Reduction Feature Selection Find a subset of the original variables (features). There are three strategies: the filter strategy, the wrapper strategy, and the embedded strategy (i.e., features are selected to add or be removed while building the model based on the prediction errors) [56][57][58][59] Feature Extraction Transforms the data in the high-dimensional space to space of fewer dimensions, which may be linear, but many nonlinear dimensionality reduction techniques also exist [60][61][62][63] Reinforcement Reinforcement Learning Q-learning Provides agents with the capability of learning to act optimally in Markovian domains by experiencing the consequences of actions, without requiring them to build maps of the domains [64][65][66][67] Temporal Difference An unsupervised technique in which the learning agent learns to predict the expected value of a variable occurring at the end of a sequence of states [68][69][70][71] Value Iteration Progressively enhancing the value function every iteration until it converges [72][73][74][75] ...
Article
Applications of machine learning (ML) methods have been used extensively to solve various complex challenges in recent years in various application areas, such as medical, financial, environmental, marketing, security, and industrial applications. ML methods are characterized by their ability to examine many data and discover exciting relationships, provide interpretation, and identify patterns. ML can help enhance the reliability, performance, predictability, and accuracy of diagnostic systems for many diseases. This survey provides a comprehensive review of the use of ML in the medical field highlighting standard technologies and how they affect medical diagnosis. Five major medical applications are deeply discussed, focusing on adapting the ML models to solve the problems in cancer, medical chemistry, brain, medical imaging, and wearable sensors. Finally, this survey provides valuable references and guidance for researchers, practitioners, and decision-makers framing future research and development directions.
... In this work, a policy is a sequence of on/off status of the HVAC system over a defined time horizon. Let s k+1 = s M (s k , x k ) describe the evolution from time step k to the next time step, k + 1, where s M is the underlying mathematical model of the studied system (see [24] for a detailed HEMS formulation). In this problem, the system model is a system of the ordinary differential equations (ODE) (1) and (2). ...
Preprint
Full-text available
This work investigates the extent to which phase change material (PCM) in the building's envelope can be used as an alternative to battery storage systems to increase self-consumption of rooftop solar photovoltaic (PV) generation. In particular, we explore the electricity cost-savings and increase in PV self-consumption that can be achieved by using PCMs and the operation of the heating, ventilation, and air conditioning (HVAC) system optimised by a home energy management system (HEMS). In more detail, we consider a HEMS with an HVAC system, rooftop PV, and a PCM layer integrated into the building envelope. The objective of the HEMS optimisation is to minimise electricity costs while maximising PV self-consumption and maintaining the indoor building temperature in a preferred comfort range. Solving this problem is challenging due to PCM's nonlinear characteristics, and using methods that can deal with the resulting non-convexity of the optimisation problem, like dynamic programming is computationally expensive. Therefore, we use multi-timescale approximate dynamic programming (MADP) that we developed in our earlier work to explore a number of Australian PCM scenarios. Specifically, we analyse a large number of residential buildings across five Australian capital cities. We find that using PCM can reduce annual electricity costs by between 10.6 % in Brisbane and 19 % in Adelaide. However, somewhat surprisingly, using PCM reduces PV self-consumption by between 1.5 % in Brisbane and 2.7 % in Perth.
... SHEM system provide an efficient service to monitor and control electricity storage, generation and consumption. Smart home contain different sensors through which different sensing data is collected from appliances, this data will communicate through HAN,s (Home area network) for real time monitoring and different operational mode are perform from personal laptop or mobile [13]. In this regard different techniques are used. ...
Preprint
Full-text available
Instead of planting new electricity generation units, there is a need to design an efficient energy management system to achieve a normalized trend of power consumption. Smart grid has been evolved as a solution, where Demand Response (DR) strategy is used to modify the nature of demand of consumer. In return, utility pay incentives to the consumer. The increasing load demand in residential area and irregular electricity load profile have encouraged us to propose an efficient Home Energy Management System (HEMS) for optimal scheduling of home appliances. In order to meet the electricity demand of the consumers, the energy consumption pattern of a consumer is maintained through scheduling the appliances in day-ahead and real-time bases. In this paper we propose a hybrid algorithm Bacterial foraging Ant colony optimization is proposed (HB-ACO) which contain both BFA and ACO properties. Primary objectives of scheduling is to shift load from On-peak hour to Off-peak hours to reduce electricity cost and peak to average ratio. A comparison of these algorithms is also presented in terms of performance parameters electricity cost, reduction of PAR and user comfort in term of waiting time. The proposed techniques are evaluated using two pricing scheme time of use and critical peak pricing. The HB-ACO shows better performance as compared to ACO and BFA which is evident from the simulation results Moreover the concept of coordination among home appliances is presented for real time scheduling. We consider this is knapsack problem and solve it through Ant colony optimization algorithm.
... A lookup tables based ADP algorithm for the real-time energy management of the MG under uncertainties is proposed in [10], while the optimal operation of the MG is also formulated as a stochastic nonlinear programming problem. In [11], they use the ADP method to solve a smart home energy management system and improve the performance of such problem by considering uncertainties of the system. A stochastic gaspower network constrained unit commitment model considering both combined-cycle units and gas Network is established in [12], in order to prevent the curse of dimensionality they also use ADP method to deal with the stochastic control problem. ...
... Let C min and C max be the minimum and maximum values of S ER , ∆P max Load , ∆P max RES and ∆P max G be the maximum power variation constraints of ∆P Load , ∆P RES and ∆P G , respectively. The optimal control problem is then formulated as (11). ...
Article
The functionality of energy routing among microgrids is becoming increasingly important with the progress of deploying smart power systems all over the world. For higher energy routing performance and better renewable energy integration, a new type of electrical device, called energy router (ER), is being developed as a part of the infrastructure of the future energy Internet (EI). Generally, the long-term operation of ERs requires an effective energy management scheme for the energy storage inside these devices. In this paper, considering the randomness of power generation by renewable energy sources and the stochastic power usage of loads in EI scenario, the compressive sensing is adopted for the solution to the nonlinear energy storage management problem which is essential for the design of ERs. The compressive sensing method used in this paper is proven to be more efficient than the conventional Monte Carlo methods and polynomial chaos expansion method, and the performance of the proposed method is evaluated with numerical examples.
... Heating, ventilation, and air conditioning (HVAC) makes a significant contribution to building energy use (up to 50 % in some countries) and operating costs. Moreover, buildings contribute 20 %-40 % of overall energy use, making HVAC of growing importance in the context of demand response (DR) 1 In particular, a potential DR resource is the buildings' thermal inertia, which can be used as an energy storage system. However, lightweight buildings, which dominate the residential building stock in Australia and which are the focus of our paper, have low thermal inertia and correspondingly high HVAC costs. ...
... This task can be cast as an optimal HVAC scheduling problem, with an objective of minimizing electricity cost while maintaining the indoor temperature within a desired comfort range. In the existing literature, this type of optimization problem is classified as a home energy management (HEM) problem [1]- [3]. In spite of the ample literature on the use of PCM for improving thermal performance of buildings [4]- [10], there is a palpable lack of understanding on how to integrate PCM into HEM for cost minimization, where the non-linear nature of its energy storage can be exploited using suitable scheduling methods. ...
... To bridge this gap, and in contrast to much of the literature on HEM [1]- [3], we consider HEM that consists of an HVAC system as a controllable device and a PCM layer as an energy storage system. To date, most HEM optimization problems are solved using linear programming or mixed integer linear programming. ...
Conference Paper
Full-text available
This paper considers energy management in buildings with phase change material (PCM) that serves as a thermal energy storage system. In this setting, the optimal scheduling of an HVAC system is challenging because of the nonlinear characteristics of the PCM, which makes solving the corresponding optimization problem using conventional optimization techniques impractical. Instead, we propose a novel approximate dynamic programming (ADP) methodology to reduce the computational burden, while maintaining the quality of the solution. Specifically, the method incorporates multi-timescale Markov decision processes and a neural network function approximator of the state transition model, coupled with an underlying statespace approximation. The method is demonstrated on an energy management problem for a typical building in Sydney, Australia, over a year. The results demonstrate that the proposed method performs well with a computational speed-up of up to 157,600 times compared to the direct application of DP.
... In [16], a policy-improvement method is proposed, which is warm-started by assuming a "good" policy learned by experience. In [17], a battery is controlled using Approximate dynamic programming. Dynamic programming methods is a standard approach for tackling MDPs; however, the curse of dimensionality prevents this family of methods from generalizing to problems with multiple agents (e.g., EVs), unless simplifying assumptions are made. ...
... and solving problem (17), where in constraint (4), it is set ...
Article
In modern power systems, distributed energy resources (DERs) are considered a valuable source of flexibility towards accommodating high penetration of Renewable Energy Sources (RES). In this paper we consider an economic dispatch problem for a community of DERs, where energy management decisions are made online and under uncertainty. We model multiple sources of uncertainty such as RES, wholesale electricity prices as well as the arrival times and energy needs of a set of Electric Vehicles. The economic dispatch problem is formulated as a multi-agent Markov Decision Process. The difficulties lie in the curse of dimensionality and in guaranteeing the satisfaction of constraints under uncertainty. A novel method, that combines duality and deep learning, is proposed to tackle these challenges. In particular, a Neural Network (NN) is trained to return the optimal dual variables of the economic dispatch problem. By training the NN on the dual problem instead of the primal, the number of output neurons is dramatically reduced, which enhances the performance and reliability of the NN. Finally, by treating the resulting dual variables as prices, each DER can self-schedule, which guarantees the satisfaction of its constraints. Our simulations show that the proposed scheme performs reliably and efficiently.
... In addition to turning on/off domestic loads, these devices provide customer consumption information, and, in case of insufficient supply, these plugs disconnect non-priority loads, thus minimizing energy waste. An energy management system was used by (Keerthisinghe, Verbic, & Chapman, 2018) to reduce the coincident usage of appliances with higher average power. Technical and operational restrictions related to household appliances were taken into account to achieve energy savings. ...
Article
Demand response (DR) programs based on pricing options allow residential customers to achieve a financial reduction in their energy bill due to changes in their consumption patterns, especially during peak periods. However, when a large number of consumers adopt this energy management program, the demand shifted to periods with low energy prices can generate new demand peaks. As a result, the quality of the power supply service may be compromised. To address this concern, this paper proposes a mixed-integer linear programming (MILP) model that aims to improve the load factor (LF) related to the demand profile of customers. To achieve this goal, an intelligent scheduling strategy for household appliances that considers flexibility in customer comfort, here called customer hourly preferences, is developed. Based on these preferences, the strategy seeks the efficient daily usage of smart appliances, mainly those with higher average power, to avoid its coincident consumption in periods with lower energy rates, thus mitigating the appearance of new peaks. In the proposed model, the operating expenses of both customers and the electricity company (ECO) are minimized. A set of technical and operational constraints such as the average power, number of times utilized, and average time of usage of home appliances, as well as the charging rate, average time for charging, and initial state-of-charge (SoC) of the plug-in electric vehicle (PEV) battery, are considered. Uncertainties related to the periods of the day when a given appliance (including PEV) is turned-on for consumption are modeled using a Monte Carlo Method (MCM). The MILP model is solved using a commercial solver CPLEX that makes use of classical optimization techniques to ensure the optimal solution to this problem. The performance of the MILP model was tested through two case studies. Case study 1 considers a group of consumers with the same income, while case study 2 triples the number of consumers in the previous case considering different incomes. The results show the importance of the proposed tool for analyzing and evaluating prospective scenarios that guarantee the efficient usage of electric energy with the lowest financial expense for both consumers and the ECO.
... An optimal scheduling scheme proposed for users by approximating as Markov perfect equilibrium of a fully observable stochastic game using online load scheduling based on actor-critic RL approach for online optimal scheduling of DR appliances [18]. An approximate dynamic programming (ADP) based approach with temporal difference learning used for effective scheduling of household appliance's uncertain electricity [19]. A new real-time incentive-based DR algorithm using RL and deep neural network (DNN) applied to balance energy demand and supply by smart grid [20]. ...
Article
Full-text available
Demand response modelling have paved an important role in smart grid at a greater perspective. DR analysis exhibits the analysis of scheduling of appliances for an optimal strategy at the user’s side with an effective pricing scheme. In this proposed work, the entire model is done in three different steps. The first step develops strategy patterns for the users considering integration of renewable energy and effective demand response analysis is done. The second step in the process exhibits the learning process of the consumers using Robust Adversarial Reinforcement Learning for privacy process among the users. The third step develops optimal strategy plan for the users for maintaining privacy among the users. Considering the uncertainties of the user’s behavioral patterns, typical pricing schemes are involved with integration of renewable energy at the user’ side so that an optimal strategy is obtained. The optimal strategy for scheduling the appliances solving privacy issues and considering renewable energy at user’ side is done using Robust Adversarial Reinforcement learning and Gradient Based Nikaido-Isoda Function which gives an optimal accuracy. The results of the proposed work exhibit optimal strategy plan for the users developing proper learning paradigm. The effectiveness of the proposed work with mathematical modelling are validated using real time data and shows the demand response strategy plan with proper learning access model. The results obtained among the set of strategy develops 80 % of the patterns created with the learning paradigm moves with optimal DR scheduling patterns. This work embarks the best learning DR pattern created for the future set of consumers following the strategy so privacy among the users can be maintained effectively.