Survey on predictive representations of state

WANG Li,GAO Yang,WANG Wei-wei

2010-01-01

Abstract:Predictive state representations(PSR) is a novel method for modeling discrete dynamic systems.PSR represents the system′s state using a set of prediction vectors of action-observation sequence.It surveyed basic concepts,problems,methods and latest extended applications of PSR,and points out the development future of the PSR model.

What problem does this paper attempt to address?

Survey of Predictive State Representations

雷珠,刘峰,赵志宏

DOI: https://doi.org/10.3969/j.issn.1001-3695.2010.02.001

2010-01-01

Abstract:Predictive state representations(PSRs) are new models for discrete-time finite action and observation stochastic systems.Because a PSR represents the system's state as a set of predictions of the observable outcomes of tests performed in the system,it can solve the computing problems in exist stochastic decision systems.This paper introduced the principles of PSR models,surveyed the PSR model and planning techniques,analyzed and compared the fundamental principles behind the modeling and planning algorithms of PSR,pointed out the development trend,and gave the challenges that the research of PSR was facing.
Predictive Models in Software Engineering: Challenges and Opportunities

Yanming Yang,Xin Xia,David Lo,Tingting Bi,John Grundy,Xiaohu Yang

DOI: https://doi.org/10.1145/3503509

IF: 3.685

2022-01-01

ACM Transactions on Software Engineering and Methodology

Abstract:Predictive models are one of the most important techniques that are widely applied in many areas of software engineering. There have been a large number of primary studies that apply predictive models and that present well-performed studies in various research domains, including software requirements, software design and development, testing and debugging, and software maintenance. This article is a first attempt to systematically organize knowledge in this area by surveying a body of 421 papers on predictive models published between 2009 and 2020. We describe the key models and approaches used, classify the different models, summarize the range of key application areas, and analyze research results. Based on our findings, we also propose a set of current challenges that still need to be addressed in future work and provide a proposed research road map for these opportunities.
Extended State Space Predictive Control for a Class of Nonlinear Systems

Ri-dong ZHANG,Shu-qing WANG,Ping LI

DOI: https://doi.org/10.3321/j.issn:1001-0920.2005.07.020

2005-01-01

Abstract:An extended state space predictive control method for a class of discrete nonlinear systems is presented by extending the nonlinear model and by using the local linearization of nonlinear function along the reference trajectory. The structure of this method is similar to classic PI optimal controller and possesses a feed-forward control of future P-step setpoints. It considers both the state variations and the output errors in the cost function and thus is superior to methods which only consider output errors. Simulation results show its efficiency.
Hilbert Space Embeddings of Predictive State Representations

Byron Boots,Geoffrey Gordon,Arthur Gretton

DOI: https://doi.org/10.48550/arXiv.1309.6819

2013-09-26

Abstract:Predictive State Representations (PSRs) are an expressive class of models for controlled stochastic processes. PSRs represent state as a set of predictions of future observable events. Because PSRs are defined entirely in terms of observable data, statistically consistent estimates of PSR parameters can be learned efficiently by manipulating moments of observed training data. Most learning algorithms for PSRs have assumed that actions and observations are finite with low cardinality. In this paper, we generalize PSRs to infinite sets of observations and actions, using the recent concept of Hilbert space embeddings of distributions. The essence is to represent the state as a nonparametric conditional embedding operator in a Reproducing Kernel Hilbert Space (RKHS) and leverage recent work in kernel methods to estimate, predict, and update the representation. We show that these Hilbert space embeddings of PSRs are able to gracefully handle continuous actions and observations, and that our learned models outperform competing system identification algorithms on several prediction benchmarks.

Machine Learning
Provably Efficient UCB-type Algorithms For Learning Predictive State Representations

Ruiquan Huang,Yingbin Liang,Jing Yang

2024-02-07

Abstract:The general sequential decision-making problem, which includes Markov decision processes (MDPs) and partially observable MDPs (POMDPs) as special cases, aims at maximizing a cumulative reward by making a sequence of decisions based on a history of observations and actions over time. Recent studies have shown that the sequential decision-making problem is statistically learnable if it admits a low-rank structure modeled by predictive state representations (PSRs). Despite these advancements, existing approaches typically involve oracles or steps that are computationally intractable. On the other hand, the upper confidence bound (UCB) based approaches, which have served successfully as computationally efficient methods in bandits and MDPs, have not been investigated for more general PSRs, due to the difficulty of optimistic bonus design in these more challenging settings. This paper proposes the first known UCB-type approach for PSRs, featuring a novel bonus term that upper bounds the total variation distance between the estimated and true models. We further characterize the sample complexity bounds for our designed UCB-type algorithms for both online and offline PSRs. In contrast to existing approaches for PSRs, our UCB-type algorithms enjoy computational tractability, last-iterate guaranteed near-optimal policy, and guaranteed model accuracy.

Machine Learning
Predictive-State Decoders: Encoding the Future into Recurrent Networks

Arun Venkatraman,Nicholas Rhinehart,Wen Sun,Lerrel Pinto,Martial Hebert,Byron Boots,Kris M. Kitani,J. Andrew Bagnell

DOI: https://doi.org/10.48550/arXiv.1709.08520

2017-09-25

Abstract:Recurrent neural networks (RNNs) are a vital modeling technique that rely on internal states learned indirectly by optimization of a supervised, unsupervised, or reinforcement training loss. RNNs are used to model dynamic processes that are characterized by underlying latent states whose form is often unknown, precluding its analytic representation inside an RNN. In the Predictive-State Representation (PSR) literature, latent state processes are modeled by an internal state representation that directly models the distribution of future observations, and most recent work in this area has relied on explicitly representing and targeting sufficient statistics of this probability distribution. We seek to combine the advantages of RNNs and PSRs by augmenting existing state-of-the-art recurrent neural networks with Predictive-State Decoders (PSDs), which add supervision to the network's internal state representation to target predicting future observations. Predictive-State Decoders are simple to implement and easily incorporated into existing training pipelines via additional loss regularization. We demonstrate the effectiveness of PSDs with experimental results in three different domains: probabilistic filtering, Imitation Learning, and Reinforcement Learning. In each, our method improves statistical performance of state-of-the-art recurrent baselines and does so with fewer iterations and less data.

Machine Learning
State Sequences Prediction Via Fourier Transform for Representation Learning

Mingxuan Ye,Yufei Kuang,Jie Wang,Rui Yang,Wengang Zhou,Houqiang Li,Feng Wu

2023-01-01

Abstract:While deep reinforcement learning (RL) has been demonstrated effective in solving complex control tasks, sample efficiency remains a key challenge due to the large amounts of data required for remarkable performance. Existing research explores the application of representation learning for data-efficient RL, e.g., learning predictive representations by predicting long-term future states. However, many existing methods do not fully exploit the structural information inherent in sequential state signals, which can potentially improve the quality of long-term decision-making but is difficult to discern in the time domain. To tackle this problem, we propose State Sequences Prediction via Fourier Transform (SPF), a novel method that exploits the frequency domain of state sequences to extract the underlying patterns in time series data for learning expressive representations efficiently. Specifically, we theoretically analyze the existence of structural information in state sequences, which is closely related to policy performance and signal regularity, and then propose to predict the Fourier transform of infinite-step future state sequences to extract such information. One of the appealing features of SPF is that it is simple to implement while not requiring storage of infinite-step future states as prediction targets. Experiments demonstrate that the proposed method outperforms several state-of-the-art algorithms in terms of both sample efficiency and performance.
An Overview of Natural Language State Representation for Reinforcement Learning

Brielen Madureira,David Schlangen

DOI: https://doi.org/10.48550/arXiv.2007.09774

2020-07-20

Abstract:A suitable state representation is a fundamental part of the learning process in Reinforcement Learning. In various tasks, the state can either be described by natural language or be natural language itself. This survey outlines the strategies used in the literature to build natural language state representations. We appeal for more linguistically interpretable and grounded representations, careful justification of design decisions and evaluation of the effectiveness of different approaches.

Computation and Language
Status quo and prospect of power system state estimation

Bijun Li,Yusheng Xue,Jinwen Gu,Zhenxiang Han

1998-01-01

Abstract:State estimation is an important part of advanced software in modern EMS. This paper reviews the status quo of state estimation. including the principle of state estimation. the application of WLS estimator. disposition of bad data. external network. analysis of observability. optimal measurement placement and the extension of state estimation. Some important problems to be studied further for state estimation are proposed.
Deep Representation Learning for Dynamical Systems Modeling

Anna Shalova,Ivan Oseledets

DOI: https://doi.org/10.48550/arXiv.2002.05111

2020-02-10

Abstract:Proper states' representations are the key to the successful dynamics modeling of chaotic systems. Inspired by recent advances of deep representations in various areas such as natural language processing and computer vision, we propose the adaptation of the state-of-art Transformer model in application to the dynamical systems modeling. The model demonstrates promising results in trajectories generation as well as in the general attractors' characteristics approximation, including states' distribution and Lyapunov exponent.

Dynamical Systems,Machine Learning,Numerical Analysis,Chaotic Dynamics,Computational Physics
Bridging State and History Representations: Understanding Self-Predictive RL

Tianwei Ni,Benjamin Eysenbach,Erfan Seyedsalehi,Michel Ma,Clement Gehring,Aditya Mahajan,Pierre-Luc Bacon

2024-04-21

Abstract:Representations are at the core of all deep reinforcement learning (RL) methods for both Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). Many representation learning methods and theoretical frameworks have been developed to understand what constitutes an effective representation. However, the relationships between these methods and the shared properties among them remain unclear. In this paper, we show that many of these seemingly distinct methods and frameworks for state and history abstractions are, in fact, based on a common idea of self-predictive abstraction. Furthermore, we provide theoretical insights into the widely adopted objectives and optimization, such as the stop-gradient technique, in learning self-predictive representations. These findings together yield a minimalist algorithm to learn self-predictive representations for states and histories. We validate our theories by applying our algorithm to standard MDPs, MDPs with distractors, and POMDPs with sparse rewards. These findings culminate in a set of preliminary guidelines for RL practitioners.

Machine Learning,Artificial Intelligence
Towards Learning Controllable Representations of Physical Systems

Kevin Haninger,Raul Vicente Garcia,Joerg Krueger

DOI: https://doi.org/10.48550/arXiv.2011.09906

IF: 5.414

2020-11-16

Machine Learning

Abstract:Learned representations of dynamical systems reduce dimensionality, potentially supporting downstream reinforcement learning (RL). However, no established methods predict a representation's suitability for control and evaluation is largely done via downstream RL performance, slowing representation design. Towards a principled evaluation of representations for control, we consider the relationship between the true state and the corresponding representations, proposing that ideally each representation corresponds to a unique true state. This motivates two metrics: temporal smoothness and high mutual information between true state/representation. These metrics are related to established representation objectives, and studied on Lagrangian systems where true state, information requirements, and statistical properties of the state can be formalized for a broad class of systems. These metrics are shown to predict reinforcement learning performance in a simulated peg-in-hole task when comparing variants of autoencoder-based representations.
A Survey on Event Prediction Methods from a Systems Perspective: Bringing Together Disparate Research Areas

Janik-Vasily Benzin,Stefanie Rinderle-Ma

DOI: https://doi.org/10.48550/arXiv.2302.04018

IF: 14.4

2023-02-08

Artificial Intelligence

Abstract:Event prediction is the ability of anticipating future events, i.e., future real-world occurrences, and aims to support the user in deciding on actions that change future events towards a desired state. An event prediction method learns the relation between features of past events and future events. It is applied to newly observed events to predict corresponding future events that are evaluated with respect to the user's desired future state. If the predicted future events do not comply with this state, actions are taken towards achieving desirable future states. Evidently, event prediction is valuable in many application domains such as business and natural disasters. The diversity of application domains results in a diverse range of methods that are scattered across various research areas which, in turn, use different terminology for event prediction methods. Consequently, sharing methods and knowledge for developing future event prediction methods is restricted. To facilitate knowledge sharing on account of a comprehensive classification, integration, and assessment of event prediction methods, we combine taxonomies and take a systems perspective to integrate event prediction methods into a single system, elicit requirements and assess existing work with respect to the requirements. Based on the assessment, we identify open challenges and discuss future research directions.
Intrinsic Dynamics-Driven Generalizable Scene Representations for Vision-Oriented Decision-Making Applications

Dayang Liang,Jinyang Lai,Yunlong Liu

2024-06-30

Abstract:How to improve the ability of scene representation is a key issue in vision-oriented decision-making applications, and current approaches usually learn task-relevant state representations within visual reinforcement learning to address this problem. While prior work typically introduces one-step behavioral similarity metrics with elements (e.g., rewards and actions) to extract task-relevant state information from observations, they often ignore the inherent dynamics relationships among the elements that are essential for learning accurate representations, which further impedes the discrimination of short-term similar task/behavior information in long-term dynamics transitions. To alleviate this problem, we propose an intrinsic dynamics-driven representation learning method with sequence models in visual reinforcement learning, namely DSR. Concretely, DSR optimizes the parameterized encoder by the state-transition dynamics of the underlying system, which prompts the latent encoding information to satisfy the state-transition process and then the state space and the noise space can be distinguished. In the implementation and to further improve the representation ability of DSR on encoding similar tasks, sequential elements' frequency domain and multi-step prediction are adopted for sequentially modeling the inherent dynamics. Finally, experimental results show that DSR has achieved significant performance improvements in the visual Distracting DMControl control tasks, especially with an average of 78.9\% over the backbone baseline. Further results indicate that it also achieves the best performances in real-world autonomous driving applications on the CARLA simulator. Moreover, qualitative analysis results validate that our method possesses the superior ability to learn generalizable scene representations on visual tasks. The source code is available at <a class="link-external link-https" href="https://github.com/DMU-XMU/DSR" rel="external noopener nofollow">this https URL</a>.

Artificial Intelligence
State Space Models on Temporal Graphs: A First-Principles Study

Jintang Li,Ruofan Wu,Xinzhou Jin,Boqun Ma,Liang Chen,Zibin Zheng

2024-10-29

Abstract:Over the past few years, research on deep graph learning has shifted from static graphs to temporal graphs in response to real-world complex systems that exhibit dynamic behaviors. In practice, temporal graphs are formalized as an ordered sequence of static graph snapshots observed at discrete time points. Sequence models such as RNNs or Transformers have long been the predominant backbone networks for modeling such temporal graphs. Yet, despite the promising results, RNNs struggle with long-range dependencies, while transformers are burdened by quadratic computational complexity. Recently, state space models (SSMs), which are framed as discretized representations of an underlying continuous-time linear dynamical system, have garnered substantial attention and achieved breakthrough advancements in independent sequence modeling. In this work, we undertake a principled investigation that extends SSM theory to temporal graphs by integrating structural information into the online approximation objective via the adoption of a Laplacian regularization term. The emergent continuous-time system introduces novel algorithmic challenges, thereby necessitating our development of GraphSSM, a graph state space model for modeling the dynamics of temporal graphs. Extensive experimental results demonstrate the effectiveness of our GraphSSM framework across various temporal graph benchmarks.

Machine Learning,Artificial Intelligence
State-wise Safe Reinforcement Learning: A Survey

Weiye Zhao,Tairan He,Rui Chen,Tianhao Wei,Changliu Liu

2023-07-01

Abstract:Despite the tremendous success of Reinforcement Learning (RL) algorithms in simulation environments, applying RL to real-world applications still faces many challenges. A major concern is safety, in another word, constraint satisfaction. State-wise constraints are one of the most common constraints in real-world applications and one of the most challenging constraints in Safe RL. Enforcing state-wise constraints is necessary and essential to many challenging tasks such as autonomous driving, robot manipulation. This paper provides a comprehensive review of existing approaches that address state-wise constraints in RL. Under the framework of State-wise Constrained Markov Decision Process (SCMDP), we will discuss the connections, differences, and trade-offs of existing approaches in terms of (i) safety guarantee and scalability, (ii) safety and reward performance, and (iii) safety after convergence and during training. We also summarize limitations of current methods and discuss potential future directions.

Machine Learning,Artificial Intelligence,Robotics
Predictive representations: building blocks of intelligence

Wilka Carvalho,Momchil S. Tomov,William de Cothi,Caswell Barry,Samuel J. Gershman

2024-07-11

Abstract:Adaptive behavior often requires predicting future events. The theory of reinforcement learning prescribes what kinds of predictive representations are useful and how to compute them. This paper integrates these theoretical ideas with work on cognition and neuroscience. We pay special attention to the successor representation (SR) and its generalizations, which have been widely applied both as engineering tools and models of brain function. This convergence suggests that particular kinds of predictive representations may function as versatile building blocks of intelligence.

Artificial Intelligence,Machine Learning
Probabilistic State-Dependent Grammars for Plan Recognition

David V. Pynadath,Michael P. Wellman

DOI: https://doi.org/10.48550/arXiv.1301.3888

2013-01-16

Abstract:Techniques for plan recognition under uncertainty require a stochastic model of the plan-generation process. We introduce Probabilistic State-Dependent Grammars (PSDGs) to represent an agent's plan-generation process. The PSDG language model extends probabilistic context-free grammars (PCFGs) by allowing production probabilities to depend on an explicit model of the planning agent's internal and external state. Given a PSDG description of the plan-generation process, we can then use inference algorithms that exploit the particular independence properties of the PSDG language to efficiently answer plan-recognition queries. The combination of the PSDG language model and inference algorithms extends the range of plan-recognition domains for which practical probabilistic inference is possible, as illustrated by applications in traffic monitoring and air combat.

Artificial Intelligence
State representation modeling for deep reinforcement learning based recommendation

Feng Liu,Ruiming Tang,Xutao Li,Weinan Zhang,Yunming Ye,Haokun Chen,Huifeng Guo,Yuzhou Zhang,Xiuqiang He

DOI: https://doi.org/10.1016/j.knosys.2020.106170

2020-10-01

Abstract:<p>Reinforcement learning techniques have recently been introduced to interactive recommender systems to capture the dynamic patterns of user behavior during the interaction with recommender systems and perform planning to optimize long-term performance. Most existing research work focuses on designing policy and learning algorithms of the recommender agent but seldom cares about the state representation of the environment, which is indeed essential for the recommendation decision making. In this paper, we first formulate the interactive recommender system problem with a deep reinforcement learning recommendation framework. Within this framework, we then carefully design four state representation schemes for learning the recommendation policy. Inspired by recent advances in feature interaction modeling in user response prediction, we discover that explicitly modeling user-item interactions in state representation can largely help the recommendation policy perform effective reinforcement learning. Extensive experiments on four real-world datasets are conducted under both the offline and simulated online evaluation settings. The experimental results demonstrate the proposed state representation schemes lead to better performance over the state-of-the-art methods.</p>

computer science, artificial intelligence
Unsupervised Representation Learning in Partially Observable Atari Games

Li Meng,Morten Goodwin,Anis Yazidi,Paal Engelstad

DOI: https://doi.org/10.48550/arXiv.2303.07437

IF: 5.414

2023-03-13

Machine Learning

Abstract:State representation learning aims to capture latent factors of an environment. Contrastive methods have performed better than generative models in previous state representation learning research. Although some researchers realize the connections between masked image modeling and contrastive representation learning, the effort is focused on using masks as an augmentation technique to represent the latent generative factors better. Partially observable environments in reinforcement learning have not yet been carefully studied using unsupervised state representation learning methods. In this article, we create an unsupervised state representation learning scheme for partially observable states. We conducted our experiment on a previous Atari 2600 framework designed to evaluate representation learning models. A contrastive method called Spatiotemporal DeepInfomax (ST-DIM) has shown state-of-the-art performance on this benchmark but remains inferior to its supervised counterpart. Our approach improves ST-DIM when the environment is not fully observable and achieves higher F1 scores and accuracy scores than the supervised learning counterpart. The mean accuracy score averaged over categories of our approach is ~66%, compared to ~38% of supervised learning. The mean F1 score is ~64% to ~33%.

Survey on predictive representations of state

Survey of Predictive State Representations

Predictive Models in Software Engineering: Challenges and Opportunities

Extended State Space Predictive Control for a Class of Nonlinear Systems

Hilbert Space Embeddings of Predictive State Representations

Provably Efficient UCB-type Algorithms For Learning Predictive State Representations

Predictive-State Decoders: Encoding the Future into Recurrent Networks

State Sequences Prediction Via Fourier Transform for Representation Learning

An Overview of Natural Language State Representation for Reinforcement Learning

Status quo and prospect of power system state estimation

Deep Representation Learning for Dynamical Systems Modeling

Bridging State and History Representations: Understanding Self-Predictive RL

Towards Learning Controllable Representations of Physical Systems

A Survey on Event Prediction Methods from a Systems Perspective: Bringing Together Disparate Research Areas

Intrinsic Dynamics-Driven Generalizable Scene Representations for Vision-Oriented Decision-Making Applications

State Space Models on Temporal Graphs: A First-Principles Study

State-wise Safe Reinforcement Learning: A Survey

Predictive representations: building blocks of intelligence

Probabilistic State-Dependent Grammars for Plan Recognition

State representation modeling for deep reinforcement learning based recommendation

Unsupervised Representation Learning in Partially Observable Atari Games