SCRL: Self-supervised Continual Reinforcement Learning for Domain Adaptation
Yuyang Fang,Bin Guo,Jiaqi Liu,Kaixing Zhao,Yasan Ding,Na Wang,Zhiwen Yu
DOI: https://doi.org/10.1109/aiotsys58602.2023.00030
2023-01-01
Abstract:Deep reinforcement learning outperforms humans in many tasks. However, in reality, the scenarios of agent deployment often change, and the performance of models may therefore degrade, resulting in unreasonable decisions in new scenarios. Therefore, it is important for the model to own the evolution ability in new scenarios. Although recent attempts have studied the continual training of models through reward signals or self-supervised learning, on the one hand, previously used reward signals in the target domain are generally difficult to obtain and on the other hand, the performance of self-supervised evolution is always limited. In addition, adapting to the target domain usually results in forgetting the source domain. To overcome the above problems, in this paper, we propose a self-supervised continual reinforcement learning method, called SCRL, which combines reinforcement learning tasks with self-supervised learning auxiliary tasks and weight regularizes. Specifically, in the source domain, we first jointly train different objectives of reinforcement and self-supervised learning, which enables to share the same encoder. The Fisher information matrix is then calculated to record the importance of the model parameters to the source domain. In the target domain, the encoder is used for the reinforcement learning task and is updated by self-supervised learning under the control of the Fisher regularizer. Our extensive experiments on four continuous control tasks from the DeepMind Control suite showed that in the absence of reward signals, the proposed SCRL can effectively adapt the model to the target domain without catastrophically forgetting the source domain.