Abstract:Reinforcement learning (RL) shows the potential to address drawbacks of rule-based control and model predictive control and exhibits great effectiveness in heating, ventilation and air conditioning (HVAC) systems. Most studies employed model-free RL to achieve building energy conservation and increase indoor comfort. However, model-free RL algorithms face the challenge of sample efficiency which causes long-time training and restricts their applications. Model-based RL is considered an alternative avenue for accelerating learning and promoting the application of RL, but it also has limitations due to modeling approaches and accuracy. In addition, few studies propose model-based RL algorithms and investigate performance gaps between model-free and model-based RL in HVAC systems. Therefore, this study conducts a comprehensive performance comparison between model-free and model-based RL to identify the current issues with RL control in HVAC systems. The open-source building optimization testing (BOPTEST) framework is employed as the virtual environment to evaluate the control performance and computational burden. Then Dueling Deep Q-Networks and Soft Actor-Critic are developed, and a state-of-the-art model-based RL framework is employed to develop their model-based versions. The comparison results showed that all RL controllers outperform the baseline control in terms of indoor temperature and operation costs. Model-based RL can achieve a control performance as good as model-free RL with a shorter training time based on its high sample efficiency. Moreover, due to massive and quickly generated data, model-based RL can accelerate the learning of RL agents, though the model is inaccurate at the early training stage. This study would provide some insights into the RL control selection and improvements in HVAC systems.

8rl-RBM and BZF-PLC based mathematical control of passive solar heating/cooling of building

A Multiple Model Approach for Predictive Control of Indoor Thermal Environment with High Resolution

Explicit model predictive control of split-type air conditioning system

Constrained Reinforcement Learning for Safe Heat Pump Control

Data-driven control of micro-climate in buildings: An event-triggered reinforcement learning approach

A Safe and Data-efficient Model-based Reinforcement Learning System for HVAC Control

Reinforcement Learning Control Strategy for Differential Pressure Setpoint in Large-scale Multi-source Looped District Cooling System

Reinforcement Learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use

Prediction of outlet air characteristics and thermal performance of a symmetrical solar air heater via machine learning to develop a model-based operational control scheme—an experimental study

Experimental analysis of data-driven control for a building heating system

Real building implementation of a deep reinforcement learning controller to enhance energy efficiency and indoor temperature control

PV-Optimized Heat Pump Control in Multi-Family Buildings Using a Reinforcement Learning Approach

MBRL-MC: An HVAC Control Approach via Combining Model-based Deep Reinforcement Learning and Model Predictive Control

Multi-zone HVAC Control with Model-Based Deep Reinforcement Learning

Cooling Water System Control of a Commercial Complex Based on Reinforcement Learning

Experimental study of model predictive control for an air-conditioning system with dedicated outdoor air system

Energy efficient model based algorithm for control of building HVAC systems

Data-driven control of room temperature and bidirectional EV charging using deep reinforcement learning: Simulations and experiments

Hybrid-Model-Based Deep Reinforcement Learning for Heating, Ventilation, and Air-Conditioning Control

Comparative study of model-based and model-free reinforcement learning control performance in HVAC systems

Deep Reinforcement Learning for Heat Pump Control