To Switch or Not to Switch? Balanced Policy Switching in Offline Reinforcement Learning

Tao Ma,Xuzhi Yang,Zoltan Szabo

2024-07-02

Abstract:Reinforcement learning (RL) -- finding the optimal behaviour (also referred to as policy) maximizing the collected long-term cumulative reward -- is among the most influential approaches in machine learning with a large number of successful applications. In several decision problems, however, one faces the possibility of policy switching -- changing from the current policy to a new one -- which incurs a non-negligible cost (examples include the shifting of the currently applied educational technology, modernization of a computing cluster, and the introduction of a new webpage design), and in the decision one is limited to using historical data without the availability for further online interaction. Despite the inevitable importance of this offline learning scenario, to our best knowledge, very little effort has been made to tackle the key problem of balancing between the gain and the cost of switching in a flexible and principled way. Leveraging ideas from the area of optimal transport, we initialize the systematic study of policy switching in offline RL. We establish fundamental properties and design a Net Actor-Critic algorithm for the proposed novel switching formulation. Numerical experiments demonstrate the efficiency of our approach on multiple benchmarks of the Gymnasium.

Machine Learning,Information Theory

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily addresses the following two core issues:

1. **Policy Switching in Offline Reinforcement Learning**: In offline reinforcement learning, how to balance potential gains and switching costs when transitioning from an old policy to a new one? Specifically, after generating a large amount of offline data using an old policy over a long period, there are now only short-term opportunities to execute the policy, and switching policies incurs non-negligible costs. The core of the research is how to rigorously define the policy switching problem in such a scenario and design algorithms under the new problem formulation to find better policies.

2. **Constructing Flexible and Expressive Switching Cost Functions**: Existing switching cost functions mainly focus on online learning scenarios and usually consider only two simple forms of costs—global costs and local costs. These cost functions lack flexibility and expressiveness, failing to describe the degree of difference between two policies. Therefore, the paper proposes a new family of cost functions based on optimal transport theory. This family of cost functions not only includes previously defined global and local costs as special cases but also measures the differences between different policies more finely, thereby better reflecting the switching costs in practical applications.

Through the above two aspects of work, the paper proposes a new policy switching problem and establishes an algorithm called "Net Actor-Critic" (NAC) to find the optimal switching policy. Experimental results show that this method performs well in multiple benchmarks.