Abstract:The rapid urbanization and increasing use of distributed renewable energy resources have imposed a significant burden on power networks. Smart buildings equipped with artificial intelligence technology can play a pivotal role in energy management, ultimately enhancing energy efficiency and voltage quality. However, ensuring voltage stability within large-scale smart building systems presents challenges due to the coexistence of diverse energy sources and the fluctuating nature of renewable energy. This paper proposes a safe multi-energy management framework achieved by online decentralized execution and centralized training for large scale smart buildings in distribution networks. The energy management problem is formulated as a safety-augmented Markov decision process, presenting intractability for dynamic programming due to its extensive continuous state space. To solve this issue and improve the convergence speed and training process stability, a safety-augmented constrained multi-agent reinforcement learning algorithm based on reward extrapolation is proposed. In this algorithm, hazard values are introduced to enhance non-safe multi-agent reinforcement learning algorithms and meet safety constraints. A novel reward network is designed by imitating expert underlying intentions to ensure the rationality of the reward function for multi-objective tasks. Additionally, the loss function for estimating the Q-network is redesigned during training process to guarantee effective convergence. Theoretical analysis is conducted to provide the convergence guarantee. Numerical case studies based on actual data are performed to validate the effectiveness and scalability of our approach, showing that smart buildings can achieve superior energy management performance while ensuring voltage safety for distribution networks. The source code of the proposed algorithm will be available at https://github.com/SYiyun/CMARL-EX.

Sinergym: a building simulation and control framework for training reinforcement learning agents

SINERGYM -- A virtual testbed for building energy optimization with Reinforcement Learning

BEAR: Physics-Principled Building Environment for Control and Reinforcement Learning

Applications in CityLearn Gym Environment for Multi-Objective Control Benchmarking in Grid-Interactive Buildings and Districts

A Lightweight Calibrated Simulation Enabling Efficient Offline Learning for Optimal Control of Real Buildings

Synconn_build: A python based synthetic dataset generator for testing and validating control-oriented neural networks for building dynamics prediction

Real-World Data and Calibrated Simulation Suite for Offline Training of Reinforcement Learning Agents to Optimize Energy and Emission in Buildings for Environmental Sustainability

Exploring Deep Reinforcement Learning for Holistic Smart Building Control

Reinforcement learning building control approach harnessing imitation learning

PhysQ: A Physics Informed Reinforcement Learning Framework for Building Control

SDGym: Low-Code Reinforcement Learning Environments using System Dynamics Models

A high-fidelity building performance simulation test bed for the development and evaluation of advanced controls

Real-world challenges for multi-agent reinforcement learning in grid-interactive buildings

Energy Management Based on Safe Multi-Agent Reinforcement Learning for Smart Buildings in Distribution Networks

Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms

Data-driven control of micro-climate in buildings: An event-triggered reinforcement learning approach

Grid-Interactive Multi-Zone Building Control Using Reinforcement Learning with Global-Local Policy Search

Simulation-based Model Learning for Optimization of Building Energy Management

Simion Zoo: A Workbench for Distributed Experimentation with Reinforcement Learning for Continuous Control Tasks

CityLearn v2: Energy-flexible, resilient, occupant-centric, and carbon-aware management of grid-interactive communities

MetaEMS: A Meta Reinforcement Learning-based Control Framework for Building Energy Management System