Abstract:This paper investigates multi-objective reinforcement learning (MORL), which focuses on learning Pareto optimal policies in the presence of multiple reward functions. Despite MORL's significant empirical success, there is still a lack of satisfactory understanding of various MORL optimization targets and efficient learning algorithms. Our work offers a systematic analysis of several optimization targets to assess their abilities to find all Pareto optimal policies and controllability over learned policies by the preferences for different objectives. We then identify Tchebycheff scalarization as a favorable scalarization method for MORL. Considering the non-smoothness of Tchebycheff scalarization, we reformulate its minimization problem into a new min-max-max optimization problem. Then, for the stochastic policy class, we propose efficient algorithms using this reformulation to learn Pareto optimal policies. We first propose an online UCB-based algorithm to achieve an $\varepsilon$ learning error with an $\tilde{\mathcal{O}}(\varepsilon^{-2})$ sample complexity for a single given preference. To further reduce the cost of environment exploration under different preferences, we propose a preference-free framework that first explores the environment without pre-defined preferences and then generates solutions for any number of preferences. We prove that it only requires an $\tilde{\mathcal{O}}(\varepsilon^{-2})$ exploration complexity in the exploration phase and demands no additional exploration afterward. Lastly, we analyze the smooth Tchebycheff scalarization, an extension of Tchebycheff scalarization, which is proved to be more advantageous in distinguishing the Pareto optimal policies from other weakly Pareto optimal policies based on entry values of preference vectors. Furthermore, we extend our algorithms and theoretical analysis to accommodate this optimization target.

Cooperative and Distributed Multiobjective Optimization for Heterogeneous Belief Rule Base

Cooperative Coevolution with Knowledge-based Dynamic Variable Decomposition for Bilevel Multiobjective Optimization

A novel optimization method for belief rule base expert system with activation rate

Multi-Objective Collaborative Optimization Based on Evolutionary Algorithms

Expensive Multi-Objective Bayesian Optimization Based on Diffusion Models

Bi-level Multi-objective Evolutionary Learning: A Case Study on Multi-task Graph Neural Topology Search

Online Updating Belief-Rule-Base Using the RIMER Approach

Multi-agent Black-box Optimization using a Bayesian Approach to Alternating Direction Method of Multipliers

Multi-Agent Bayesian Optimization with Coupled Black-Box and Affine Constraints

A structure optimization method for extended belief-rule-based classification system

Evolving Multimodal Robot Behavior via Many Stepping Stones with the Combinatorial Multiobjective Evolutionary Algorithm

Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning

Cooperated Bayesian Algorithm for Distributed Scheduling Problem

Hybridization of evolutionary algorithm and deep reinforcement learning for multi-objective orienteering optimization

CURO: Curriculum Learning for Relative Overgeneralization

A Cooperative Evolutionary Framework Based on an Improved Version of Directed Weight Vectors for Constrained Multiobjective Optimization With Deceptive Constraints

Pareto Front-Diverse Batch Multi-Objective Bayesian Optimization

Combinatorial-hybrid Optimization for Multi-agent Systems under Collaborative Tasks

A reinforcement learning approach for dynamic multi-objective optimization

A greedy belief rule base generation and learning method for classification problem

Deep Reinforcement Learning for Multiobjective Optimization