Abstract:We propose Gnu-RL: a novel approach that enables real-world deployment of reinforcement learning (RL) for building control and requires no prior information other than historical data from existing Heating Ventilation and Air Conditioning (HVAC) controllers. In contrast with existing RL agents, which are opaque to expert interrogation and need ample training to achieve reasonable control performance, Gnu-RL is much more attractive for real-world applications. Furthermore, Gnu-RL avoids the need to develop and calibrate simulation environments for each building or system under control, thus making it highly scalable. To achieve this, we bootstrap the Gnu-RL agent with domain knowledge and expert demonstration. Specifically, Gnu-RL adopts a recently-developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynamics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, enabling it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using a policy gradient algorithm. We evaluate Gnu-RL in both simulation studies and a real-world testbed. Firstly, we validated the Gnu-RL agent is indeed practical and scalable by applying it to three commercial reference buildings in simulation, and demonstrated the superiority of the Differentiable MPC policy compared to a generic neural network. In another simulation experiment, our approach saved 6.6% of the total energy compared to the best published RL result for the same environment, while maintaining a higher level of occupant comfort. Finally, Gnu-RL was deployed to control the airflow to a real-world conference room with a Variable-Air-Volume (VAV) system during a 3-weeks period. Our results show that Gnu-RL reduced cooling demand by 16.7% compared to the existing controller and tracked the temperature set-point better.

MBRL-MC: An HVAC Control Approach via Combining Model-based Deep Reinforcement Learning and Model Predictive Control

Multi-zone HVAC Control with Model-Based Deep Reinforcement Learning

Hybrid-Model-Based Deep Reinforcement Learning for Heating, Ventilation, and Air-Conditioning Control

A Safe and Data-efficient Model-based Reinforcement Learning System for HVAC Control

A Multiple Model Approach for Predictive Control of Indoor Thermal Environment with High Resolution

Model Predictive Control Strategy Using Encoder-Decoder Recurrent Neural Networks for Smart Control of Thermal Environment

Modelling building HVAC control strategies using a deep reinforcement learning approach

Multi-source transfer learning method for enhancing the deployment of deep reinforcement learning in multi-zone building HVAC control

Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning

MF^2: Model-free Reinforcement Learning for Modeling-Free Building HVAC Control with Data-Driven Environment Construction in a Residential Building

Deep Reinforcement Learning for Heat Pump Control

Towards self-learning control of HVAC systems with the consideration of dynamic occupancy patterns: Application of model-free deep reinforcement learning

Reinforcement Learning Control Strategy for Differential Pressure Setpoint in Large-scale Multi-source Looped District Cooling System

Energy Consumption Optimization for Heating, Ventilation and Air Conditioning Systems Based on Deep Reinforcement Learning

A Community Energy Management Method Based on Explainable Reinforcement Learning

Comparative study of model-based and model-free reinforcement learning control performance in HVAC systems

The integration of Model Predictive Control and deep Reinforcement Learning for efficient thermal control in thermoforming processes

Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings

Gnu-RL: A Practical and Scalable Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy

A Comparison of Classical and Deep Reinforcement Learning Methods for HVAC Control

Heating ventilation air‐conditioner system for multi‐regional commercial buildings based on deep reinforcement learning