Abstract:Cooling system plays a critical role in a modern data center (DC). Developing an optimal control policy for DC cooling system is a challenging task. The prevailing approaches often rely on approximating system models that are built upon the knowledge of mechanical cooling, electrical and thermal management, which is difficult to design and may lead to sub-optimal or unstable performances. In this paper, we propose utilizing the large amount of monitoring data in DC to optimize the control policy. To do so, we cast the cooling control policy design into an energy cost minimization problem with temperature constraints, and tap it into the emerging deep reinforcement learning (DRL) framework. Specifically, we propose an end-to-end cooling control algorithm (CCA) that is based on the actor-critic framework and an off-policy offline version of the deep deterministic policy gradient (DDPG) algorithm. In the proposed CCA, an evaluation network is trained to predict an energy cost counter penalized by the cooling status of the DC room, and a policy network is trained to predict optimized control settings when gave the current load and weather information. The proposed algorithm is evaluated on the EnergyPlus simulation platform and on a real data trace collected from the National Super Computing Centre (NSCC) of Singapore. Our results show that the proposed CCA can achieve about 11% cooling cost saving on the simulation platform compared with a manually configured baseline control algorithm. In the trace-based study, we propose a de-underestimation validation mechanism as we cannot directly test the algorithm on a real DC. Even though with DUE the results are conservative, we can still achieve about 15% cooling energy saving on the NSCC data trace if we set the inlet temperature threshold at 26.6 degree Celsius.

Residual Physics and Post-Posed Shielding for Safe Deep Reinforcement Learning Method

Exploring the Vulnerability of Deep Reinforcement Learning-based Emergency Control for Low Carbon Power Systems

AdapSafe: Adaptive and Safe-Certified Deep Reinforcement Learning-Based Frequency Control for Carbon-Neutral Power Systems.

Deep Reinforcement Learning Based Approach for Active Power Security Correction Control of Power System

SafeCool: Safe and Energy-Efficient Cooling Management in Data Centers with Model-Based Reinforcement Learning

District Cooling System Control for Providing Operating Reserve Based on Safe Deep Reinforcement Learning

Comparison of Deep Reinforcement Learning Algorithms in Data Center Cooling Management: A Case Study

Physics-Constrained Vulnerability Assessment of Deep Reinforcement Learning-Based SCOPF

Physical Deep Reinforcement Learning Towards Safety Guarantee

Real-Time Optimal Power Flow Method Via Safe Deep Reinforcement Learning Based on Primal-Dual and Prior Knowledge Guidance

Hybrid-Model-Based Deep Reinforcement Learning for Heating, Ventilation, and Air-Conditioning Control

Safe Reinforcement Learning for Emergency LoadShedding of Power Systems

A Safe DRL Method for Fast Solution of Real-Time Optimal Power Flow

Adaptive Power System Emergency Control using Deep Reinforcement Learning

Mixed Deep Reinforcement Learning Considering Discrete-continuous Hybrid Action Space for Smart Home Energy Management

Physics-model-guided Worst-case Sampling for Safe Reinforcement Learning

Deep Reinforcement Learning Control for Pulsed Power Load Online Deployment in DC Shipboard Integrated Power System

Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings

Large-Scale Data Center Cooling Control Via Sample-Efficient Reinforcement Learning

Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning

Hierarchical Framework for Interpretable and Probabilistic Model-Based Safe Reinforcement Learning