The Hidden Information State model

YoungSteve,GašićMilica,KeizerSimon,MairesseFrançois,SchatzmannJost,ThomsonBlaise,YuKai

IF: 3.252

2010-01-01

Computer Speech & Language

Abstract:This paper explains how Partially Observable Markov Decision Processes (POMDPs) can provide a principled mathematical framework for modelling the inherent uncertainty in spoken dialogue systems. It...

What problem does this paper attempt to address?

The Hidden Information State Model: A Practical Framework for POMDP-based Spoken Dialogue Management

Steve Young,Milica Gasic,Simon Keizer,Francois Mairesse,Jost Schatzmann,Blaise Thomson,Kai Yu

DOI: https://doi.org/10.1016/j.csl.2009.04.001

IF: 3.252

2010-01-01

Computer Speech & Language

Abstract:This paper explains how Partially Observable Markov Decision Processes (POMDPs) can provide a principled mathematical framework for modelling the inherent uncertainty in spoken dialogue systems. It briefly summarises the basic mathematics and explains why exact optimisation is intractable. It then describes in some detail a form of approximation called the Hidden Information State model which does scale and which can be used to build practical systems. A prototype HIS system for the tourist information domain is evaluated and compared with a baseline MDP system using both user simulations and a live user trial. The results give strong support to the central contention that the POMDP-based framework is both a tractable and powerful approach to building more robust spoken dialogue systems.
Modelling User Behaviour In The His-Pomdp Dialogue Manager

S. Keizer,M. Gasic,F. Mairesse,B. Thomson,K. Yu,S. Young

DOI: https://doi.org/10.1109/SLT.2008.4777855

2008-01-01

Abstract:In the design of spoken dialogue systems that are robust to speech recognition and interpretation errors, modelling uncertainty is crucial. Recently, Partially Observable Markov Decision Processes (POMDPs) have been shown to provide a well-founded probabilistic framework for developing such systems. This paper reports on the design and evaluation of the user act model (UAM) as part of the Hidden Information State (HIS) POMDP dialogue manager. Within this system, the UAM represents the probability of a user producing a certain dialogue act, given the last system act and the dialogue state. Its design is domain-independent and founded on the notions of adjacency pairs and dialogue act preconditions. Experimental evaluation results on both simulated and real data show that the UAM plays a significant role in improving robustness, but it requires that the N-best lists of user act hypotheses and their confidence scores are of good quality.
Training and Evaluation of the HIS POMDP Dialogue System in Noise.

Milica Gasic,Simon Keizer,François Mairesse,Jost Schatzmann,Blaise Thomson,Kai Yu,Steve J. Young

DOI: https://doi.org/10.3115/1622064.1622087

2008-01-01

Abstract:This paper investigates the claim that a dialogue manager modelled as a Partially Observable Markov Decision Process (POMDP) can achieve improved robustness to noise compared to conventional state-based dialogue managers. Using the Hidden Information State (HIS) POMDP dialogue manager as an exemplar, and an MDP-based dialogue manager as a baseline, evaluation results are presented for both simulated and real dialogues in a Tourist Information Domain. The results on the simulated data show that the inherent ability to model uncertainty, allows the POMDP model to exploit alternative hypotheses from the speech understanding system. The results obtained from a user trial show that the HIS system with a trained policy performed significantly better than the MDP baseline.
Parameter Learning for POMDP Spoken Dialogue Models.

Blaise Thomson,Filip Jurcícek,Milica Gasic,Simon Keizer,François Mairesse,Kai Yu,Steve J. Young

DOI: https://doi.org/10.1109/slt.2010.5700863

2010-01-01

Abstract:The partially observable Markov decision process (POMDP) provides a popular framework for modelling spoken dialogue. This paper describes how the expectation propagation algorithm (EP) can be used to learn the parameters of the POMDP user model. Various special probability factors applicable to this task are presented, which allow the parameters be to learned when the structure of the dialogue is complex. No annotations, neither the true dialogue state nor the true semantics of user utterances, are required. Parameters optimised using the proposed techniques are shown to improve the performance of both offline transcription experiments as well as simulated dialogue management performance.
Bridging the Gap between Partially Observable Stochastic Games and Sparse POMDP Methods

Tyler Becker,Zachary Sunberg

2024-10-29

Abstract:Many real-world decision problems involve the interaction of multiple self-interested agents with limited sensing ability. The partially observable stochastic game (POSG) provides a mathematical framework for modeling these problems, however solving a POSG requires difficult reasoning over two critical factors: (1) information revealed by partial observations and (2) decisions other agents make. In the single agent case, partially observable Markov decision process (POMDP) planning can efficiently address partial observability with particle filtering. In the multi-agent case, extensive form game solution methods account for other agent's decisions, but preclude belief approximation. We propose a unifying framework that combines POMDP-inspired state distribution approximation and game-theoretic equilibrium search on information sets. This paper lays a theoretical foundation for the approach by bounding errors due to belief approximation, and empirically demonstrates effectiveness with a numerical example. The new approach enables planning in POSGs with very large state spaces, paving the way for reliable autonomous interaction in real-world physical environments and complementing multi-agent reinforcement learning.

Computer Science and Game Theory
Information-Theoretic Opacity-Enforcement in Markov Decision Processes

Chongyang Shi,Yuheng Bu,Jie Fu

2024-05-01

Abstract:The paper studies information-theoretic opacity, an information-flow privacy property, in a setting involving two agents: A planning agent who controls a stochastic system and an observer who partially observes the system states. The goal of the observer is to infer some secret, represented by a random variable, from its partial observations, while the goal of the planning agent is to make the secret maximally opaque to the observer while achieving a satisfactory total return. Modeling the stochastic system using a Markov decision process, two classes of opacity properties are considered -- Last-state opacity is to ensure that the observer is uncertain if the last state is in a specific set and initial-state opacity is to ensure that the observer is unsure of the realization of the initial state. As the measure of opacity, we employ the Shannon conditional entropy capturing the information about the secret revealed by the observable. Then, we develop primal-dual policy gradient methods for opacity-enforcement planning subject to constraints on total returns. We propose novel algorithms to compute the policy gradient of entropy for each observation, leveraging message passing within the hidden Markov models. This gradient computation enables us to have stable and fast convergence. We demonstrate our solution of opacity-enforcement control through a grid world example.

Systems and Control
Control Theory Meets POMDPs: A Hybrid Systems Approach

Mohamadreza Ahmadi,Nils Jansen,Bo Wu,Ufuk Topcu

DOI: https://doi.org/10.48550/arXiv.1905.08095

2019-05-17

Abstract:Partially observable Markov decision processes (POMDPs) provide a modeling framework for a variety of sequential decision making under uncertainty scenarios in artificial intelligence (AI). Since the states are not directly observable in a POMDP, decision making has to be performed based on the output of a Bayesian filter (continuous beliefs). Hence, POMDPs are often computationally intractable to solve exactly and researchers resort to approximate methods often using discretizations of the continuous belief space. These approximate solutions are, however, prone to discretization errors, which has made POMDPs ineffective in applications, wherein guarantees for safety, optimality, or performance are required. To overcome the complexity challenge of POMDPs, we apply notions from control theory. The goal is to determine the reachable belief space of a POMDP, that is, the set of all possible evolutions given an initial belief distribution over the states and a set of actions and observations. We begin by casting the problem of analyzing a POMDP into analyzing the behavior of a discrete-time switched system. For estimating the reachable belief space, we find over-approximations in terms of sub-level sets of Lyapunov functions. Furthermore, in order to verify safety and optimality requirements of a given POMDP, we formulate a barrier certificate theorem, wherein we show that if there exists a barrier certificate satisfying a set of inequalities along with the belief update equation of the POMDP, the safety and optimality properties are guaranteed to hold. In both cases, we show how the calculations can be decomposed into smaller problems that can be solved in parallel. The conditions we formulate can be computationally implemented as a set of sum-of-squares programs. We illustrate the applicability of our method by addressing two problems in active ad scheduling and machine teaching.

Systems and Control
Statistical Methods for Building Robust Spoken Dialogue Systems in an Automobile

Pirros Tsiakoulis,Milica Gasic,Matthew Henderson,Joaquin Planells-Lerma,Jorge Prombonas,Blaise Thomson,Kai Yu,Steve Young,Eli Tzirkel

DOI: https://doi.org/10.1098/rsta.2000.0593

2013-01-01

Abstract:We investigate the potential of statistical techniques for spoken dialogue systems in an automotive environment. Specifically, we focus on partially observable Markov decision processes (POMDPs), which have recently been proposed as a statistical framework for building dialogue managers (DMs). These statistical DMs have explicit models of uncertainty, which allow alternative recognition hypotheses to be exploited, and dialogue management policies that can be optimised automatically using reinforcement learning. This paper presents a voice-based in-car system for providing information about local amenities (e.g. restaurants). A user trial is described which compares performance of a trained statistical dialogue manager with a conventional handcrafted system. The results demonstrate the differing behaviours of the two systems and the performance advantage obtained when using the statistical approach.
A Possibilistic Model for Qualitative Sequential Decision Problems under Uncertainty in Partially Observable Environments

Regis Sabbadin

DOI: https://doi.org/10.48550/arXiv.1301.6736

2013-01-24

Abstract:In this article we propose a qualitative (ordinal) counterpart for the Partially Observable Markov Decision Processes model (POMDP) in which the uncertainty, as well as the preferences of the agent, are modeled by possibility distributions. This qualitative counterpart of the POMDP model relies on a possibilistic theory of decision under uncertainty, recently developed. One advantage of such a qualitative framework is its ability to escape from the classical obstacle of stochastic POMDPs, in which even with a finite state space, the obtained belief state space of the POMDP is infinite. Instead, in the possibilistic framework even if exponentially larger than the state space, the belief state space remains finite.

Artificial Intelligence
User Study of the Bayesian Update of Dialogue State Approach to Dialogue Management

B. Thomson,M. Gasic,S. Keizer,E. Mairesse,J. Schatzmann,K. Yu,S. Young

DOI: https://doi.org/10.21437/interspeech.2008-81

2008-01-01

Abstract:This paper presents the results of a comparative user evaluation of various approaches to dialogue management. The major contribution is a comparison of traditional systems against a system that uses a Bayesian Update of Dialogue State approach. This approach is based on the Partially Observable Markov Decision Process (POMDP), which has previously been shown to give improved robustness in simulation experiments. Results from this paper show that the benefits demonstrated in simulation experiments are also obtained when testing a live system with real users.
Point-Based Methods for Model Checking in Partially Observable Markov Decision Processes

Maxime Bouton,Jana Tumova,Mykel J. Kochenderfer

DOI: https://doi.org/10.48550/arXiv.2001.03809

2020-01-12

Abstract:Autonomous systems are often required to operate in partially observable environments. They must reliably execute a specified objective even with incomplete information about the state of the environment. We propose a methodology to synthesize policies that satisfy a linear temporal logic formula in a partially observable Markov decision process (POMDP). By formulating a planning problem, we show how to use point-based value iteration methods to efficiently approximate the maximum probability of satisfying a desired logical formula and compute the associated belief state policy. We demonstrate that our method scales to large POMDP domains and provides strong bounds on the performance of the resulting policy.

Artificial Intelligence
A Novel Factored Pomdp Model For Affective Dialogue Management

Fuji Ren,Yu Wang,Changqin Quan

DOI: https://doi.org/10.3233/IFS-162126

2016-01-01

Abstract:Partially observable Markov decision process (POMDP) model has been demonstrated many times to be suited for robust spoken dialogue management. Recently, some factored representations of POMDP model are designed for specific dialogue tasks. This paper proposes a novel factored POMDP model to describe a new application of affective dialogue management. Different from existing models, the user's state space and the system's observation space are both divided into two distinct components: goal and emotion. Moreover, the system's action space is for the first time factored into two parts, i.e., goal response and emotion response, and the reward function is accordingly updated by weighted sum of the two-part rewards. An example of intelligent music player is given to explain how to apply the new model to build an affective dialogue system. Four experiments are designed to reveal the influence of key parameters on the system performance. The simulation results demonstrate the rationality and feasibility of the proposed model.
Partially Observable Markov Decision Processes in Robotics: A Survey

Mikko Lauri,David Hsu,Joni Pajarinen

DOI: https://doi.org/10.1109/TRO.2022.3200138

2022-09-21

Abstract:Noisy sensing, imperfect control, and environment changes are defining characteristics of many real-world robot tasks. The partially observable Markov decision process (POMDP) provides a principled mathematical framework for modeling and solving robot decision and control tasks under uncertainty. Over the last decade, it has seen many successful applications, spanning localization and navigation, search and tracking, autonomous driving, multi-robot systems, manipulation, and human-robot interaction. This survey aims to bridge the gap between the development of POMDP models and algorithms at one end and application to diverse robot decision tasks at the other. It analyzes the characteristics of these tasks and connects them with the mathematical and algorithmic properties of the POMDP framework for effective modeling and solution. For practitioners, the survey provides some of the key task characteristics in deciding when and how to apply POMDPs to robot tasks successfully. For POMDP algorithm designers, the survey provides new insights into the unique challenges of applying POMDPs to robot systems and points to promising new directions for further research.

Robotics,Artificial Intelligence,Systems and Control
Gaussian processes for fast policy optimisation of POMDP-based dialogue managers

M. Gašić,F. Jurčíček,S. Keizer,F. Mairesse,B. Thomson,K. Yu,S. Young

2010-01-01

Abstract:Modelling dialogue as a Partially Observable Markov Decision Process (POMDP) enables a dialogue policy robust to speech understanding errors to be learnt. However, a major challenge in POMDP policy learning is to maintain tractability, so the use of approximation is inevitable. We propose applying Gaussian Processes in Reinforcement learning of optimal POMDP dialogue policies, in order (1) to make the learning process faster and (2) to obtain an estimate of the uncertainty of the approximation. We first demonstrate the idea on a simple voice mail dialogue task and then apply this method to a real-world tourist information dialogue task.
Back-off Action Selection in Summary Space-Based POMDP Dialogue Systems.

M. Gasic,F. Lefevre,F. Jurcicek,S. Keizer,F. Mairesse,B. Thomson,K. Yu,S. Young

DOI: https://doi.org/10.1109/asru.2009.5373416

2009-01-01

Abstract:This paper deals with the issue of invalid state-action pairs in the Partially Observable Markov Decision Process (POMDP) framework, with a focus on real-world tasks where the need for approximate solutions exacerbates this problem. In particular, when modelling dialogue as a POMDP, both the state and the action space must be reduced to smaller scale summary spaces in order to make learning tractable. However, since not all actions are valid in all states, the action proposed by the policy in summary space sometimes leads to an invalid action when mapped back to master space. Some form of back-off scheme must then be used to generate an alternative action. This paper demonstrates how the value function derived during reinforcement learning can be used to order back-off actions in an N-best list. Compared to a simple baseline back-off strategy and to a strategy that extends the summary space to minimise the occurrence of invalid actions, the proposed N-best action selection scheme is shown to be significantly more robust.
Monte Carlo Sampling Methods for Approximating Interactive POMDPs

Prashant Doshi,Piotr J. Gmytrasiewicz

DOI: https://doi.org/10.1613/jair.2630

2014-01-15

Abstract:Partially observable Markov decision processes (POMDPs) provide a principled framework for sequential planning in uncertain single agent settings. An extension of POMDPs to multiagent settings, called interactive POMDPs (I-POMDPs), replaces POMDP belief spaces with interactive hierarchical belief systems which represent an agent's belief about the physical world, about beliefs of other agents, and about their beliefs about others' beliefs. This modification makes the difficulties of obtaining solutions due to complexity of the belief and policy spaces even more acute. We describe a general method for obtaining approximate solutions of I-POMDPs based on particle filtering (PF). We introduce the interactive PF, which descends the levels of the interactive belief hierarchies and samples and propagates beliefs at each level. The interactive PF is able to mitigate the belief space complexity, but it does not address the policy space complexity. To mitigate the policy space complexity -- sometimes also called the curse of history -- we utilize a complementary method based on sampling likely observations while building the look ahead reachability tree. While this approach does not completely address the curse of history, it beats back the curse's impact substantially. We provide experimental results and chart future work.

Artificial Intelligence
Imprecise Probabilities Meet Partial Observability: Game Semantics for Robust POMDPs

Eline M. Bovy,Marnix Suilen,Sebastian Junges,Nils Jansen

2024-07-29

Abstract:Partially observable Markov decision processes (POMDPs) rely on the key assumption that probability distributions are precisely known. Robust POMDPs (RPOMDPs) alleviate this concern by defining imprecise probabilities, referred to as uncertainty sets. While robust MDPs have been studied extensively, work on RPOMDPs is limited and primarily focuses on algorithmic solution methods. We expand the theoretical understanding of RPOMDPs by showing that 1) different assumptions on the uncertainty sets affect optimal policies and values; 2) RPOMDPs have a partially observable stochastic game (POSG) semantic; and 3) the same RPOMDP with different assumptions leads to semantically different POSGs and, thus, different policies and values. These novel semantics for RPOMDPs give access to results for POSGs, studied in game theory; concretely, we show the existence of a Nash equilibrium. Finally, we classify the existing RPOMDP literature using our semantics, clarifying under which uncertainty assumptions these existing works operate.

Artificial Intelligence,Computer Science and Game Theory
Explainable Finite-Memory Policies for Partially Observable Markov Decision Processes

Muqsit Azeem,Debraj Chakraborty,Sudeep Kanav,Jan Kretinsky

2024-11-20

Abstract:Partially Observable Markov Decision Processes (POMDPs) are a fundamental framework for decision-making under uncertainty and partial observability. Since in general optimal policies may require infinite memory, they are hard to implement and often render most problems undecidable. Consequently, finite-memory policies are mostly considered instead. However, the algorithms for computing them are typically very complex, and so are the resulting policies. Facing the need for their explainability, we provide a representation of such policies, both (i) in an interpretable formalism and (ii) typically of smaller size, together yielding higher explainability. To that end, we combine models of Mealy machines and decision trees; the latter describing simple, stationary parts of the policies and the former describing how to switch among them. We design a translation for policies of the finite-state-controller (FSC) form from standard literature and show how our method smoothly generalizes to other variants of finite-memory policies. Further, we identify specific properties of recently used "attractor-based" policies, which allow us to construct yet simpler and smaller representations. Finally, we illustrate the higher explainability in a few case studies.

Artificial Intelligence,Machine Learning,Robotics,Systems and Control
A Framework for Sequential Planning in Multi-Agent Settings

P. Doshi,P. J. Gmytrasiewicz

DOI: https://doi.org/10.1613/jair.1579

2011-09-10

Abstract:This paper extends the framework of partially observable Markov decision processes (POMDPs) to multi-agent settings by incorporating the notion of agent models into the state space. Agents maintain beliefs over physical states of the environment and over models of other agents, and they use Bayesian updates to maintain their beliefs over time. The solutions map belief states to actions. Models of other agents may include their belief states and are related to agent types considered in games of incomplete information. We express the agents autonomy by postulating that their models are not directly manipulable or observable by other agents. We show that important properties of POMDPs, such as convergence of value iteration, the rate of convergence, and piece-wise linearity and convexity of the value functions carry over to our framework. Our approach complements a more traditional approach to interactive settings which uses Nash equilibria as a solution paradigm. We seek to avoid some of the drawbacks of equilibria which may be non-unique and do not capture off-equilibrium behaviors. We do so at the cost of having to represent, process and continuously revise models of other agents. Since the agents beliefs may be arbitrarily nested, the optimal solutions to decision making problems are only asymptotically computable. However, approximate belief updates and approximately optimal plans are computable. We illustrate our framework using a simple application domain, and we show examples of belief updates and value functions.

Artificial Intelligence,Multiagent Systems
Prospective Side Information for Latent MDPs

Jeongyeol Kwon,Yonathan Efroni,Shie Mannor,Constantine Caramanis

2023-10-11

Abstract:In many interactive decision-making settings, there is latent and unobserved information that remains fixed. Consider, for example, a dialogue system, where complete information about a user, such as the user's preferences, is not given. In such an environment, the latent information remains fixed throughout each episode, since the identity of the user does not change during an interaction. This type of environment can be modeled as a Latent Markov Decision Process (LMDP), a special instance of Partially Observed Markov Decision Processes (POMDPs). Previous work established exponential lower bounds in the number of latent contexts for the LMDP class. This puts forward a question: under which natural assumptions a near-optimal policy of an LMDP can be efficiently learned? In this work, we study the class of LMDPs with {\em prospective side information}, when an agent receives additional, weakly revealing, information on the latent context at the beginning of each episode. We show that, surprisingly, this problem is not captured by contemporary settings and algorithms designed for partially observed environments. We then establish that any sample efficient algorithm must suffer at least $\Omega(K^{2/3})$-regret, as opposed to standard $\Omega(\sqrt{K})$ lower bounds, and design an algorithm with a matching upper bound.

Machine Learning,Information Theory

The Hidden Information State model

The Hidden Information State Model: A Practical Framework for POMDP-based Spoken Dialogue Management

Modelling User Behaviour In The His-Pomdp Dialogue Manager

Training and Evaluation of the HIS POMDP Dialogue System in Noise.

Parameter Learning for POMDP Spoken Dialogue Models.

Bridging the Gap between Partially Observable Stochastic Games and Sparse POMDP Methods

Information-Theoretic Opacity-Enforcement in Markov Decision Processes

Control Theory Meets POMDPs: A Hybrid Systems Approach

Statistical Methods for Building Robust Spoken Dialogue Systems in an Automobile

A Possibilistic Model for Qualitative Sequential Decision Problems under Uncertainty in Partially Observable Environments

User Study of the Bayesian Update of Dialogue State Approach to Dialogue Management

Point-Based Methods for Model Checking in Partially Observable Markov Decision Processes

A Novel Factored Pomdp Model For Affective Dialogue Management

Partially Observable Markov Decision Processes in Robotics: A Survey

Gaussian processes for fast policy optimisation of POMDP-based dialogue managers

Back-off Action Selection in Summary Space-Based POMDP Dialogue Systems.

Monte Carlo Sampling Methods for Approximating Interactive POMDPs

Imprecise Probabilities Meet Partial Observability: Game Semantics for Robust POMDPs

Explainable Finite-Memory Policies for Partially Observable Markov Decision Processes

A Framework for Sequential Planning in Multi-Agent Settings

Prospective Side Information for Latent MDPs