Abstract:Text-based reinforcement learning agents have predominantly been neural network-based models with embeddings-based representation, learning uninterpretable policies that often do not generalize well to unseen games. On the other hand, neuro-symbolic methods, specifically those that leverage an intermediate formal representation, are gaining significant attention in language understanding tasks. This is because of their advantages ranging from inherent interpretability, the lesser requirement of training data, and being generalizable in scenarios with unseen data. Therefore, in this paper, we propose a modular, NEuro-Symbolic Textual Agent (NESTA) that combines a generic semantic parser with a rule induction system to learn abstract interpretable rules as policies. Our experiments on established text-based game benchmarks show that the proposed NESTA method outperforms deep reinforcement learning-based techniques by achieving better generalization to unseen test games and learning from fewer training interactions.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in text - based reinforcement learning, existing neural - network - based agents usually rely on embedding representations, the learned policies are difficult to interpret, and they have poor generalization ability on unseen games. To solve these problems, the author proposes a modular neural - symbolic text agent (NESTA), which combines a general - purpose semantic parser and a rule induction system to learn abstract and interpretable rules as policies. The main goals of NESTA are: 1. **Improve generalization ability**: By learning abstract rules, NESTA can better generalize to unseen game scenarios, rather than just relying on entities in specific training data. 2. **Enhance interpretability**: The rules learned by NESTA are symbolic and can be understood and debugged by humans, thus increasing the transparency of the model. 3. **Reduce the need for training data**: Compared with traditional deep reinforcement learning methods, NESTA requires fewer training interactions to achieve good performance. 4. **Improve sample efficiency**: NESTA shows better sample efficiency during the training process and can achieve high test performance within a fewer number of training steps. Specifically, NESTA achieves these goals through the following steps: - **Semantic parser**: Convert text observations into symbolic triples, using Abstract Meaning Representation (AMR) as an intermediate representation. - **Rule learner**: Utilize Inductive Logic Programming (ILP) to learn logical rules from reward signals. These rules abstract entities in the game, making them applicable to test games that contain unseen entities. - **Pruning module**: Remove actions that do not contribute to future rewards through a look - ahead strategy, reduce the action space, and improve search efficiency. Through these modules, NESTA shows performance superior to existing deep reinforcement learning methods in text - based reinforcement learning tasks, especially in terms of generalization ability and sample efficiency.

Learning Symbolic Rules over Abstract Meaning Representations for Textual Reinforcement Learning

End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations

Interpretable end-to-end Neurosymbolic Reinforcement Learning agents

Language Understanding for Text-based Games Using Deep Reinforcement Learning

Enhancing Text-based Reinforcement Learning Agents with Commonsense Knowledge

EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning

Learning Symbolic Rules for Interpretable Deep Reinforcement Learning

Towards Learning Abstractions via Reinforcement Learning

Learning to Play Text-based Adventure Games with Maximum Entropy Reinforcement Learning

Using reinforcement learning to learn how to play text-based games

Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networks

Abstract then Play: A Skill-centric Reinforcement Learning Framework for Text-based Games.

Large Language Models Are Neurosymbolic Reasoners

On the Effects of Fine-tuning Language Models for Text-Based Reinforcement Learning

Symbolic Task Inference in Deep Reinforcement Learning

Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search

Learning Generalizable Behavior via Visual Rewrite Rules

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision (Short Version)

LOA: Logical Optimal Actions for Text-based Interaction Games

A Neuro-Symbolic Approach to Multi-Agent RL for Interpretability and Probabilistic Decision Making

Learning Invariable Semantical Representation from Language for Extensible Policy Generalization