Development of a Soft Actor Critic Deep Reinforcement Learning Approach for Harnessing Energy Flexibility in a Large Office Building

Anjukan Kathirgamanathan,Eleni Mangina,Donal P. Finn

DOI: https://doi.org/10.1016/j.egyai.2021.100101

2021-04-25

Abstract:This research is concerned with the novel application and investigation of `Soft Actor Critic' (SAC) based Deep Reinforcement Learning (DRL) to control the cooling setpoint (and hence cooling loads) of a large commercial building to harness energy flexibility. The research is motivated by the challenge associated with the development and application of conventional model-based control approaches at scale to the wider building stock. SAC is a model-free DRL technique that is able to handle continuous action spaces and which has seen limited application to real-life or high-fidelity simulation implementations in the context of automated and intelligent control of building energy systems. Such control techniques are seen as one possible solution to supporting the operation of a smart, sustainable and future electrical grid. This research tests the suitability of the SAC DRL technique through training and deployment of the agent on an EnergyPlus based environment of the office building. The SAC DRL was found to learn an optimal control policy that was able to minimise energy costs by 9.7% compared to the default rule-based control (RBC) scheme and was able to improve or maintain thermal comfort limits over a test period of one week. The algorithm was shown to be robust to the different hyperparameters and this optimal control policy was learnt through the use of a minimal state space consisting of readily available variables. The robustness of the algorithm was tested through investigation of the speed of learning and ability to deploy to different seasons and climates. It was found that the SAC DRL requires minimal training sample points and outperforms the RBC after three months of operation and also without disruption to thermal comfort during this period. The agent is transferable to other climates and seasons although further retraining or hyperparameter tuning is recommended.

Machine Learning,Systems and Control

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to utilize the "Soft Actor Critic" (SAC) deep reinforcement learning technology to control the cooling set - point (and thus the cooling load) in large - scale office buildings in order to realize the utilization of energy flexibility. The motivation for the research lies in the challenges faced by traditional model - based control methods when applied on a large scale to a broader building stock. SAC is a model - free deep reinforcement learning technology that can handle continuous action spaces and has limited applications in the control of automatic and intelligent building energy systems in actual or high - fidelity simulation implementations. This control technology is regarded as a possible solution to support the operation of a smart and sustainable future power grid. Specifically, the paper tests the applicability of the SAC deep reinforcement learning technology by training and deploying agents in an EnergyPlus - based environment. The study finds that SAC deep reinforcement learning can learn the optimal control strategy, which can reduce energy costs by 9.7% compared with the default rule - based control (RBC) scheme and can improve or maintain the thermal comfort limit during a one - week test period. In addition, the algorithm is robust to different hyper - parameters, and this optimal control strategy is learned by using the minimum state space (including easily accessible variables). The robustness of the algorithm is also tested through the investigation of the learning speed and the deployment ability in different seasons and climates. The results show that SAC deep reinforcement learning requires the minimum number of training sample points, surpasses RBC after three months of operation, and does not cause interference to thermal comfort during this period. Although the agent can be transferred to other climates and seasons, further retraining or hyper - parameter adjustment is recommended.

Development of a Soft Actor Critic Deep Reinforcement Learning Approach for Harnessing Energy Flexibility in a Large Office Building

Real building implementation of a deep reinforcement learning controller to enhance energy efficiency and indoor temperature control

Large-Scale Data Center Cooling Control Via Sample-Efficient Reinforcement Learning

Toward Smart Multizone HVAC Control by Combining Context-Aware System and Deep Reinforcement Learning.

Exploring Deep Reinforcement Learning for Holistic Smart Building Control

Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system

Optimal Dynamic Thermal Management for Data Center Via Soft Actor-Critic Algorithm with Dynamic Control Interval and Combined-Value State Space

Energy optimization for HVAC systems in multi-VAV open offices: A deep reinforcement learning approach

Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings

Distributed Multi-Agent Deep Reinforcement Learning Framework for Whole-building HVAC Control

An experimental evaluation of deep reinforcement learning algorithms for HVAC control

SafeCool: Safe and Energy-Efficient Cooling Management in Data Centers with Model-Based Reinforcement Learning

Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning

Comparison of two deep reinforcement learning algorithms towards an optimal policy for smart building thermal control

Real-World Data and Calibrated Simulation Suite for Offline Training of Reinforcement Learning Agents to Optimize Energy and Emission in Buildings for Environmental Sustainability

Energy optimization for regional buildings based on distributed reinforcement learning

Heating ventilation air‐conditioner system for multi‐regional commercial buildings based on deep reinforcement learning

Automated deep reinforcement learning for real-time scheduling strategy of multi-energy system integrated with post-carbon and direct-air carbon captured system

Evaluating Reinforcement Learning Algorithms in Residential Energy Saving and Comfort Management