Abstract:Model-Based Reinforcement Learning (MBRL) has been widely studied for Heating, Ventilation, and Air Conditioning (HVAC) control in buildings. One of the critical challenges is the large amount of data required to effectively train neural networks for modeling building dynamics. This paper presents CLUE, an MBRL system for HVAC control in buildings. CLUE optimizes HVAC operations by integrating a Gaussian Process (GP) model to model building dynamics with uncertainty awareness. CLUE utilizes GP to predict state transitions as Gaussian distributions, effectively capturing prediction uncertainty and enhancing decision-making under sparse data conditions. Our approach employs a meta-kernel learning technique to efficiently set GP kernel hyperparameters using domain knowledge from diverse buildings. This drastically reduces the data requirements typically associated with GP models in HVAC applications. Additionally, CLUE incorporates these uncertainty estimates into a Model Predictive Path Integral (MPPI) algorithm, enabling the selection of safe, energy-efficient control actions. This uncertainty-aware control strategy evaluates and selects action trajectories based on their predicted impact on energy consumption and human comfort, optimizing operations even under uncertain conditions. Extensive simulations in a five-zone office building demonstrate that CLUE reduces the required training data from hundreds of days to just seven while maintaining robust control performance. It reduces comfort violations by an average of 12.07% compared to existing MBRL methods, without compromising on energy efficiency.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address several key challenges faced by model-based reinforcement learning (MBRL) in building heating, ventilation, and air conditioning (HVAC) control: 1. **Large Data Requirement**: Traditional MBRL methods require a large amount of data to effectively train neural networks to model building dynamics. For example, existing MB2C methods need 183 days of data to achieve reliable predictions and effective control. 2. **Prediction Uncertainty**: Under sparse data conditions, existing models often exhibit significant prediction uncertainty, leading to inaccurate predictions in uncommon states. This uncertainty can affect the safety and energy efficiency of control strategies. 3. **Model Parameter Setting**: Gaussian Process (GP) models, while capable of capturing prediction uncertainty, rely on predefined kernel functions and their hyperparameters for performance. Traditional parameter selection methods either require expert experience to manually determine or need a large amount of data for gradient descent optimization, which is often impractical in real-world applications. To address these issues, the paper proposes a new MBRL system named CLUE. CLUE optimizes HVAC control through the following approaches: - **Integration of Gaussian Process Models**: Utilizing GP models to model building dynamics and capturing prediction uncertainty by predicting state transitions as Gaussian distributions. - **Meta-Kernel Learning Technique**: Developing a new meta-kernel learning method that automatically learns optimal kernel initialization from a broad dataset of different buildings, thereby reducing the need for extensive historical data from specific buildings. - **Uncertainty-Aware Control Algorithm**: Combining the uncertainty estimation of the GP model with the Model Predictive Path Integral (MPPI) algorithm to design a two-stage uncertainty-aware control algorithm, ensuring the selection of safe and energy-efficient control actions under uncertain conditions. Through these innovations, CLUE not only significantly reduces the required training data (from hundreds of days to 7 days) but also improves control performance, reduces comfort violations (by an average of 12.07%), while maintaining energy efficiency.

A Safe and Data-efficient Model-based Reinforcement Learning System for HVAC Control

Multi-zone HVAC Control with Model-Based Deep Reinforcement Learning

Hybrid-Model-Based Deep Reinforcement Learning for Heating, Ventilation, and Air-Conditioning Control

MBRL-MC: An HVAC Control Approach via Combining Model-based Deep Reinforcement Learning and Model Predictive Control

Comparative study of model-based and model-free reinforcement learning control performance in HVAC systems

MF^2: Model-free Reinforcement Learning for Modeling-Free Building HVAC Control with Data-Driven Environment Construction in a Residential Building

Modelling building HVAC control strategies using a deep reinforcement learning approach

Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings

A Community Energy Management Method Based on Explainable Reinforcement Learning

Towards self-learning control of HVAC systems with the consideration of dynamic occupancy patterns: Application of model-free deep reinforcement learning

Successful application of predictive information in deep reinforcement learning control: A case study based on an office building HVAC system

Energy-Efficient HVAC Control Based on Reinforcement Learning and Transfer Learning in a Residential Building

Enhancing the Performance of Multi-Agent Reinforcement Learning for Controlling HVAC Systems

Energy Consumption Optimization for Heating, Ventilation and Air Conditioning Systems Based on Deep Reinforcement Learning

ED-DQN: An event-driven deep reinforcement learning control method for multi-zone residential buildings

Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning

Go Beyond Black-box Policies: Rethinking the Design of Learning Agent for Interpretable and Verifiable HVAC Control

A Comparison of Classical and Deep Reinforcement Learning Methods for HVAC Control

Transfer Learning Applied to Reinforcement Learning-Based HVAC Control

A Multi-Agent Reinforcement Learning Approach to Price and Comfort Optimization in HVAC-Systems

Laxity-Aware Scalable Reinforcement Learning for HVAC Control