Abstract:While multitask representation learning has become a popular approach in reinforcement learning (RL), theoretical understanding of why and when it works remains limited. This paper presents analyses for the statistical benefit of multitask representation learning in linear Markov Decision Process (MDP) under a generative model. In this paper, we consider an agent to learn a representation function φ out of a function class Φ from T source tasks with N data per task, and then use the learned φ̂ to reduce the required number of sample for a new task. We first discover a Least-Activated-Feature-Abundance (LAFA) criterion, denoted as κ, with which we prove that a straightforward least-square algorithm learns a policy which is Õ ( H √ C(Φ)2κd NT + κd n ) sub-optimal. Here H is the planning horizon, C (Φ) is Φ’s complexity measure, d is the dimension of the representation (usually d C(Φ)) and n is the number of samples for the new task. In multitask RL, NT is usually sufficiently large and the bound is dominated by the second term. Thus the required n is O(κdH) for the sub-optimality to be close to zero, which is much smaller than O(C(Φ)κdH) in the setting without multitask representation learning, whose sub-optimality gap is Õ ( H √ κC(Φ)2d n ) . This theoretically explains the power of multitask representation learning in reducing sample complexity. Further, we note that to ensure high sample efficiency, the LAFA criterion κ should be small. In fact, κ varies widely in magnitude depending on the different sampling distribution for new task. This indicates adaptive sampling technique is important to make κ solely depend on d. Finally, we provide empirical results of a noisy grid-world environment to corroborate our theoretical findings.

Multi-task Representation Learning for Mixed Integer Linear Programming

A Multi-Agent Learning Framework for Mixed-Integer Linear Programming

Towards Foundation Models for Mixed Integer Linear Programming

Fast and Interpretable Mixed-Integer Linear Program Solving by Learning Model Reduction

Mixed-Integer Linear Optimization via Learning-Based Two-Layer Large Neighborhood Search

Distributional MIPLIB: a Multi-Domain Library for Advancing ML-Guided MILP Methods

RL-MILP Solver: A Reinforcement Learning Approach for Solving Mixed-Integer Linear Programs with Graph Neural Networks

Multi-Task Learning with Group-Specific Feature Space Sharing

Light-MILPopt: Solving Large-scale Mixed Integer Linear Programs with Lightweight Optimizer and Small-scale Training Dataset

G2MILP: Learning to Generate Mixed-Integer Linear Programming Instances for MILP Solvers

A New Multitask Joint Learning Framework for Expensive Multi-Objective Optimization Problems

Pareto Multi-Task Learning

Multiple Task Learning Using Iteratively Reweighted Least Square.

Towards Impartial Multi-task Learning.

Multi-Task Learning as Multi-Objective Optimization

Learning Incoherent Sparse and Low-Rank Patterns from Multiple Tasks

Pre-trained Mixed Integer Optimization through Multi-variable Cardinality Branching

On the Power of Multitask Representation Learning in Linear MDP

Machine Learning Augmented Branch and Bound for Mixed Integer Linear Programming

A Multi-task Framework for Solving Multimodal Multiobjective Optimization Problems