Abstract:Dialogue policy is a crucial component in task-oriented Spoken Dialogue Systems (SDSs). As a decision function, it takes the current dialogue state as input and generates appropriate system’s response. In this paper, we explore the reinforcement learning approaches to solve this problem in an Indic language scenario. Recently, Deep Reinforcement Learning (DRL) has been used to optimise the dialogue policy. However, many DRL approaches are not sample-efficient. Hence, particular attention is given to actor-critic methods based on off-policy reinforcement learning that utilise the Experience Replay (ER) technique for reducing the bias and variance to achieve high sample efficiency. ER based actor-critic methods, such as Advantage Actor-Critic Experience Replay (A2CER) are proven to deliver competitive results in gaming environments that are fully observable and have a very small action-set. While, in SDSs, the states are not fully observable and often have to deal with the large action space. Describing the limitations of traditional methods, i.e., value-based and policy-based methods, such as high variance, low sample-efficiency, and often converging to local optima, we firstly explore the use of A2CER in dialogue policy learning. It is shown to beat the current state-of-the-art deep learning methods for SDS. Secondly, to handle the issues of early-stage performance, we utilise a demonstration corpus to pre-train the models prior to on-line policy learning. We thus experiment with the A2CER on a larger action space and find it significantly faster than the current state-of-the-art. Combining both approaches, we present a novel DRL based dialogue policy optimisation method, A2CER and its effectiveness for a task-oriented SDS in the Indic language.

On-line policy optimisation of spoken dialogue systems via live interaction with human subjects

Replicating Complex Dialogue Policy of Humans Via Offline Imitation Learning with Supervised Regularization.

Dialogue Learning with Human-in-the-Loop.

Gaussian processes for fast policy optimisation of POMDP-based dialogue managers

Learning through Dialogue Interactions by Asking Questions

Deep Reinforcement Learning for Dialogue Generation

Hyper-parameter Optimisation of Gaussian Process Reinforcement Learning for Statistical Dialogue Management.

Adversarial learning of neural user simulators for dialogue policy optimisation

Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management

Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems

Improving Interaction Quality Estimation with BiLSTMs and the Impact on Dialogue Policy Learning

Dialogue Strategy Adaptation to New Action Sets Using Multi-dimensional Modelling

Findings from Experiments of On-line Joint Reinforcement Learning of Semantic Parser and Dialogue Manager with real Users

Gaussian Process Based Deep Dyna-Q Approach for Dialogue Policy Learning.

Affordable On-line Dialogue Policy Learning

Optimizing human-interpretable dialog management policy using Genetic Algorithm

Enhancing End-to-End Multi-Task Dialogue Systems: A Study on Intrinsic Motivation Reinforcement Learning Algorithms for Improved Training and Adaptability

Experience Replay-based Deep Reinforcement Learning for Dialogue Management Optimisation

What does the User Want? Information Gain for Hierarchical Dialogue Policy Optimisation

Uncertainty Estimates for Efficient Neural Network-based Dialogue Policy Optimisation

Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems