Program of the Seventy-First Annual Meeting of the Potato Association of America at St. Paul Hotel St. Paul, Minnesota August 2–6, 1987

N.,Addy,K. Knutson,H. Sarah,Quiñones,Peter,J. Joyce,B. Mccown,P. Tovar

DOI: https://doi.org/10.1007/BF02853566

American Potato Journal

Abstract:

What problem does this paper attempt to address?

Deconstructing deep active inference

Théophile Champion,Marek Grześ,Lisa Bonheme,Howard Bowman

2023-05-08

Abstract:Active inference is a theory of perception, learning and decision making, which can be applied to neuroscience, robotics, and machine learning. Recently, reasearch has been taking place to scale up this framework using Monte-Carlo tree search and deep learning. The goal of this activity is to solve more complicated tasks using deep active inference. First, we review the existing literature, then, we progresively build a deep active inference agent. For two agents, we have experimented with five definitions of the expected free energy and three different action selection strategies. According to our experiments, the models able to solve the dSprites environment are the ones that maximise rewards. Finally, we compare the similarity of the representation learned by the layers of various agents using centered kernel alignment. Importantly, the agent maximising reward and the agent minimising expected free energy learn very similar representations except for the last layer of the critic network (reflecting the difference in learning objective), and the variance layers of the transition and encoder networks. We found that the reward maximising agent is a lot more certain than the agent minimising expected free energy. This is because the agent minimising expected free energy always picks the action down, and does not gather enough data for the other actions. In contrast, the agent maximising reward, keeps on selecting the actions left and right, enabling it to successfully solve the task. The only difference between those two agents is the epistemic value, which aims to make the outputs of the transition and encoder networks as close as possible. Thus, the agent minimising expected free energy picks a single action (down), and becomes an expert at predicting the future when selecting this action. This makes the KL divergence between the output of the transition and encoder networks small.

Artificial Intelligence
Deep Active Inference as Variational Policy Gradients

Beren Millidge

DOI: https://doi.org/10.48550/arXiv.1907.03876

2019-07-09

Abstract:Active Inference is a theory of action arising from neuroscience which casts action and planning as a bayesian inference problem to be solved by minimizing a single quantity - the variational free energy. Active Inference promises a unifying account of action and perception coupled with a biologically plausible process theory. Despite these potential advantages, current implementations of Active Inference can only handle small, discrete policy and state-spaces and typically require the environmental dynamics to be known. In this paper we propose a novel deep Active Inference algorithm which approximates key densities using deep neural networks as flexible function approximators, which enables Active Inference to scale to significantly larger and more complex tasks. We demonstrate our approach on a suite of OpenAIGym benchmark tasks and obtain performance comparable with common reinforcement learning baselines. Moreover, our algorithm shows similarities with maximum entropy reinforcement learning and the policy gradients algorithm, which reveals interesting connections between the Active Inference framework and reinforcement learning.

Machine Learning,Neural and Evolutionary Computing
A Minimal Active Inference Agent

Simon McGregor,Manuel Baltieri,Christopher L. Buckley

DOI: https://doi.org/10.48550/arXiv.1503.04187

IF: 14.4

2015-03-13

Artificial Intelligence

Abstract:Research on the so-called "free-energy principle'' (FEP) in cognitive neuroscience is becoming increasingly high-profile. To date, introductions to this theory have proved difficult for many readers to follow, but it depends mainly upon two relatively simple ideas: firstly that normative or teleological values can be expressed as probability distributions (active inference), and secondly that approximate Bayesian reasoning can be effectively performed by gradient descent on model parameters (the free-energy principle). The notion of active inference is of great interest for a number of disciplines including cognitive science and artificial intelligence, as well as cognitive neuroscience, and deserves to be more widely known. This paper attempts to provide an accessible introduction to active inference and informational free-energy, for readers from a range of scientific backgrounds. In this work introduce an agent-based model with an agent trying to make predictions about its position in a one-dimensional discretized world using methods from the FEP.
Active inference: demystified and compared

Noor Sajid,Philip J. Ball,Thomas Parr,Karl J. Friston

DOI: https://doi.org/10.1162/neco_a_01357

2020-10-31

Abstract:Active inference is a first principle account of how autonomous agents operate in dynamic, non-stationary environments. This problem is also considered in reinforcement learning (RL), but limited work exists on comparing the two approaches on the same discrete-state environments. In this paper, we provide: 1) an accessible overview of the discrete-state formulation of active inference, highlighting natural behaviors in active inference that are generally engineered in RL; 2) an explicit discrete-state comparison between active inference and RL on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of RL. We show that by operating in a pure belief-based setting, active inference agents can carry out epistemic exploration, and account for uncertainty about their environment in a Bayes-optimal fashion. Furthermore, we show that the reliance on an explicit reward signal in RL is removed in active inference, where reward can simply be treated as another observation; even in the total absence of rewards, agent behaviors are learned through preference learning. We make these properties explicit by showing two scenarios in which active inference agents can infer behaviors in reward-free environments compared to both Q-learning and Bayesian model-based RL agents; by placing zero prior preferences over rewards and by learning the prior preferences over the observations corresponding to reward. We conclude by noting that this formalism can be applied to more complex settings if appropriate generative models can be formulated. In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation, and demonstrate these behaviors in a OpenAI gym environment, alongside RL agents.

Artificial Intelligence,Quantitative Methods
Active Inference as a Model of Agency

Lancelot Da Costa,Samuel Tenka,Dominic Zhao,Noor Sajid

2024-01-24

Abstract:Is there a canonical way to think of agency beyond reward maximisation? In this paper, we show that any type of behaviour complying with physically sound assumptions about how macroscopic biological agents interact with the world canonically integrates exploration and exploitation in the sense of minimising risk and ambiguity about states of the world. This description, known as active inference, refines the free energy principle, a popular descriptive framework for action and perception originating in neuroscience. Active inference provides a normative Bayesian framework to simulate and model agency that is widely used in behavioural neuroscience, reinforcement learning (RL) and robotics. The usefulness of active inference for RL is three-fold. \emph{a}) Active inference provides a principled solution to the exploration-exploitation dilemma that usefully simulates biological agency. \emph{b}) It provides an explainable recipe to simulate behaviour, whence behaviour follows as an explainable mixture of exploration and exploitation under a generative world model, and all differences in behaviour are explicit in differences in world model. \emph{c}) This framework is universal in the sense that it is theoretically possible to rewrite any RL algorithm conforming to the descriptive assumptions of active inference as an active inference algorithm. Thus, active inference can be used as a tool to uncover and compare the commitments and assumptions of more specific models of agency.

Artificial Intelligence
Inwardly rectifying K+ currents in intermediate cells in the cochlea of gerbils: a possible contribution to the endocochlear potential

S. Takeuchi,M. Ando

DOI: https://doi.org/10.1016/S0304-3940(98)00318-8

IF: 3.197

1998-05-15

Neuroscience Letters

Abstract:
Active inference on discrete state-spaces: a synthesis

Lancelot Da Costa,Thomas Parr,Noor Sajid,Sebastijan Veselic,Victorita Neacsu,Karl Friston

DOI: https://doi.org/10.1016/j.jmp.2020.102447

2020-03-29

Abstract:Active inference is a normative principle underwriting perception, action, planning, decision-making and learning in biological or artificial agents. From its inception, its associated process theory has grown to incorporate complex generative models, enabling simulation of a wide range of complex behaviours. Due to successive developments in active inference, it is often difficult to see how its underlying principle relates to process theories and practical implementation. In this paper, we try to bridge this gap by providing a complete mathematical synthesis of active inference on discrete state-space models. This technical summary provides an overview of the theory, derives neuronal dynamics from first principles and relates this dynamics to biological processes. Furthermore, this paper provides a fundamental building block needed to understand active inference for mixed generative models; allowing continuous sensations to inform discrete representations. This paper may be used as follows: to guide research towards outstanding challenges, a practical guide on how to implement active inference to simulate experimental behaviour, or a pointer towards various in-silico neurophysiological responses that may be used to make empirical predictions.

Neurons and Cognition
Demonstrating the Continual Learning Capabilities and Practical Application of Discrete-Time Active Inference

Rithvik Prakki

2024-10-01

Abstract:Active inference is a mathematical framework for understanding how agents (biological or artificial) interact with their environments, enabling continual adaptation and decision-making. It combines Bayesian inference and free energy minimization to model perception, action, and learning in uncertain and dynamic contexts. Unlike reinforcement learning, active inference integrates exploration and exploitation seamlessly by minimizing expected free energy. In this paper, we present a continual learning framework for agents operating in discrete time environments, using active inference as the foundation. We derive the mathematical formulations of variational and expected free energy and apply them to the design of a self-learning research agent. This agent updates its beliefs and adapts its actions based on new data without manual intervention. Through experiments in changing environments, we demonstrate the agent's ability to relearn and refine its models efficiently, making it suitable for complex domains like finance and healthcare. The paper concludes by discussing how the proposed framework generalizes to other systems, positioning active inference as a flexible approach for adaptive AI.

Artificial Intelligence,Machine Learning
Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop

Martin Biehl,Christian Guckelsberger,Christoph Salge,Simón C. Smith,Daniel Polani

DOI: https://doi.org/10.3389/fnbot.2018.00045

IF: 3.493

2018-08-30

Frontiers in Neurorobotics

Abstract:Active inference is an ambitious theory that treats perception, inference, and action selection of autonomous agents under the heading of a single principle. It suggests biologically plausible explanations for many cognitive phenomena, including consciousness. In active inference, action selection is driven by an objective function that evaluates possible future actions with respect to current, inferred beliefs about the world. Active inference at its core is independent from extrinsic rewards, resulting in a high level of robustness across e.g., different environments or agent morphologies. In the literature, paradigms that share this independence have been summarized under the notion of intrinsic motivations. In general and in contrast to active inference, these models of motivation come without a commitment to particular inference and action selection mechanisms. In this article, we study if the inference and action selection machinery of active inference can also be used by alternatives to the originally included intrinsic motivation. The perception-action loop explicitly relates inference and action selection to the environment and agent memory, and is consequently used as foundation for our analysis. We reconstruct the active inference approach, locate the original formulation within, and show how alternative intrinsic motivations can be used while keeping many of the original features intact. Furthermore, we illustrate the connection to universal reinforcement learning by means of our formalism. Active inference research may profit from comparisons of the dynamics induced by alternative intrinsic motivations. Research on intrinsic motivations may profit from an additional way to implement intrinsically motivated agents that also share the biological plausibility of active inference.

robotics,computer science, artificial intelligence,neurosciences
On efficient computation in active inference

Aswin Paul,Noor Sajid,Lancelot Da Costa,Adeel Razi

DOI: https://doi.org/10.1016/j.eswa.2024.124315

IF: 8.5

2024-06-02

Expert Systems with Applications

Abstract:Biological agents demonstrate a remarkable proficiency in calibrating appropriate scales of planning and evaluation when interacting with their environments. It follows logically that any decision-making algorithm aspiring to neurobiological plausibility must mirror these attributes, particularly regarding computational expenditure and the intricacy of evaluative processes. However, active inference encounters notable challenges in simulating apt behaviours within complex environments. These stem chiefly from its substantial computational demands and the intricate task of defining the agent's behaviour preference. We address these through a two-fold approach. First, we introduce a planning algorithm by using the Bellman-optimality principle to minimise the planning cost function (i.e., expected free energy). Briefly, we recursively compute the expected free energy of actions in reverse temporal sequence to significantly reduce the computational complexity. Secondly, inspired by the Z-learning algorithm, we propose a novel method to learn time-constrained agent preferences. We face-validate the efficacy of these through grid-world simulations and demonstrate precise model learning and planning, even under uncertainty. These algorithmic advances create new opportunities for various applications—in neuroscience and machine learning.

computer science, artificial intelligence,engineering, electrical & electronic,operations research & management science
On Predictive planning and counterfactual learning in active inference

Aswin Paul,Takuya Isomura,Adeel Razi

2024-03-19

Abstract:Given the rapid advancement of artificial intelligence, understanding the foundations of intelligent behaviour is increasingly important. Active inference, regarded as a general theory of behaviour, offers a principled approach to probing the basis of sophistication in planning and decision-making. In this paper, we examine two decision-making schemes in active inference based on 'planning' and 'learning from experience'. Furthermore, we also introduce a mixed model that navigates the data-complexity trade-off between these strategies, leveraging the strengths of both to facilitate balanced decision-making. We evaluate our proposed model in a challenging grid-world scenario that requires adaptability from the agent. Additionally, our model provides the opportunity to analyze the evolution of various parameters, offering valuable insights and contributing to an explainable framework for intelligent decision-making.

Artificial Intelligence,Machine Learning,Methodology
Active Inference on the Edge: A Design Study

Boris Sedlak,Victor Casamayor Pujol,Praveen Kumar Donta,Schahram Dustdar

2023-11-18

Abstract:Machine Learning (ML) is a common tool to interpret and predict the behavior of distributed computing systems, e.g., to optimize the task distribution between devices. As more and more data is created by Internet of Things (IoT) devices, data processing and ML training are carried out by edge devices in close proximity. To ensure Quality of Service (QoS) throughout these operations, systems are supervised and dynamically adapted with the help of ML. However, as long as ML models are not retrained, they fail to capture gradual shifts in the variable distribution, leading to an inaccurate view of the system state. Moreover, as the prediction accuracy decreases, the reporting device should actively resolve uncertainties to improve the model's precision. Such a level of self-determination could be provided by Active Inference (ACI) -- a concept from neuroscience that describes how the brain constantly predicts and evaluates sensory information to decrease long-term surprise. We encompassed these concepts in a single action-perception cycle, which we implemented for distributed agents in a smart manufacturing use case. As a result, we showed how our ACI agent was able to quickly and traceably solve an optimization problem while fulfilling QoS requirements.

Systems and Control,Artificial Intelligence,Distributed, Parallel, and Cluster Computing,Machine Learning
Artificial enactive inference in three-dimensional world

Olivier L. Georgeon,David Lurie,Paul Robertson

DOI: https://doi.org/10.1016/j.cogsys.2024.101234

IF: 4.541

2024-04-08

Cognitive Systems Research

Abstract:The theory of Enactive Inference was proposed by Karl Friston and his colleagues to explain how the brain infers knowledge about the world through the subject's interactive experiences. Sensorimotor states induce perturbations in neural activity, and the brain infers hypothetical causes in the world that may explain these perturbations. This article aims to reconcile this neuroscience theory with computer science and artificial-intelligence theories wherein artificial agents receive input data derived from the environment's state and infer internal data structures used to guide decisions. Two critical challenges arise in both the agent's active role and the inference algorithm's scalability as the environment's complexity increases. To address these challenges, we formalize artificial enactive inference through a new Spatial Enactive Markov Decision Process (SEMDP) model. This model rests on low-level control loops enacted in a three-dimensional Euclidean space containing objects. Based on the SEMDP, we present a proof-of-concept cognitive architecture and an experiment to demonstrate the transcription of the theory of enactive inference into the domain of artificial intelligence and robotics.

psychology, experimental,neurosciences,computer science, artificial intelligence
Learning and embodied decisions in active inference

Matteo Priorelli,Ivilin Peev Stoianov,Giovanni Pezzulo

DOI: https://doi.org/10.1101/2024.08.18.608439

2024-08-21

Abstract:Biological organisms constantly face the necessity to act timely in dynamic environments and balance choice accuracy against the risk of missing valid opportunities. As formalized by embodied decision models, this might require brain architectures wherein decision-making and motor control interact reciprocally, in stark contrast to traditional models that view them as serial processes. Previous studies have assessed that embodied decision dynamics emerge naturally under active inference - a computational paradigm that considers action and perception as subject to the same imperative of free energy minimization. In particular, agents can infer their targets by using their own movements (and not only external sensations) as evidence, i.e., via self-evidencing. Such models have shown that under appropriate conditions, action-generated feedback can stabilize and improve decision processes. However, how adaptation of internal models to environmental contingencies influences embodied decisions is yet to be addressed. To shed light on this challenge, in this study we systematically investigate the learning dynamics of an embodied model of decision-making during a two-alternative forced choice task, using a hybrid (discrete and continuous) active inference framework. Our results show that active inference agents can adapt to embodied contexts by learning various statistical regularities of the task - namely, prior preferences for the correct target, cue validity, and response strategies that prioritize faster or slower (but more accurate) decisions. Crucially, these results illustrate the efficacy of learning discrete preferences and strategies using sensorimotor feedback from continuous dynamics.

Neuroscience
Active inference, Bayesian optimal design, and expected utility

Noor Sajid,Lancelot Da Costa,Thomas Parr,Karl Friston

DOI: https://doi.org/10.48550/arXiv.2110.04074

IF: 5.414

2021-09-21

Machine Learning

Abstract:Active inference, a corollary of the free energy principle, is a formal way of describing the behavior of certain kinds of random dynamical systems that have the appearance of sentience. In this chapter, we describe how active inference combines Bayesian decision theory and optimal Bayesian design principles under a single imperative to minimize expected free energy. It is this aspect of active inference that allows for the natural emergence of information-seeking behavior. When removing prior outcomes preferences from expected free energy, active inference reduces to optimal Bayesian design, i.e., information gain maximization. Conversely, active inference reduces to Bayesian decision theory in the absence of ambiguity and relative risk, i.e., expected utility maximization. Using these limiting cases, we illustrate how behaviors differ when agents select actions that optimize expected utility, expected information gain, and expected free energy. Our T-maze simulations show optimizing expected free energy produces goal-directed information-seeking behavior while optimizing expected utility induces purely exploitive behavior and maximizing information gain engenders intrinsically motivated behavior.
Learning in Hybrid Active Inference Models

Poppy Collis,Ryan Singh,Paul F Kinghorn,Christopher L Buckley

2024-09-02

Abstract:An open problem in artificial intelligence is how systems can flexibly learn discrete abstractions that are useful for solving inherently continuous problems. Previous work in computational neuroscience has considered this functional integration of discrete and continuous variables during decision-making under the formalism of active inference (Parr, Friston & de Vries, 2017; Parr & Friston, 2018). However, their focus is on the expressive physical implementation of categorical decisions and the hierarchical mixed generative model is assumed to be known. As a consequence, it is unclear how this framework might be extended to learning. We therefore present a novel hierarchical hybrid active inference agent in which a high-level discrete active inference planner sits above a low-level continuous active inference controller. We make use of recent work in recurrent switching linear dynamical systems (rSLDS) which implement end-to-end learning of meaningful discrete representations via the piecewise linear decomposition of complex continuous dynamics (Linderman et al., 2016). The representations learned by the rSLDS inform the structure of the hybrid decision-making agent and allow us to (1) specify temporally-abstracted sub-goals in a method reminiscent of the options framework, (2) lift the exploration into discrete space allowing us to exploit information-theoretic exploration bonuses and (3) `cache' the approximate solutions to low-level problems in the discrete planner. We apply our model to the sparse Continuous Mountain Car task, demonstrating fast system identification via enhanced exploration and successful planning through the delineation of abstract sub-goals.

Artificial Intelligence,Systems and Control
Interactive inference: a multi-agent model of cooperative joint actions

Domenico Maisto,Francesco Donnarumma,Giovanni Pezzulo

DOI: https://doi.org/10.1109/TSMC.2023.3312585

2022-12-11

Abstract:We advance a novel computational model of multi-agent, cooperative joint actions that is grounded in the cognitive framework of active inference. The model assumes that to solve a joint task, such as pressing together a red or blue button, two (or more) agents engage in a process of interactive inference. Each agent maintains probabilistic beliefs about the goal of the joint task (e.g., should we press the red or blue button?) and updates them by observing the other agent's movements, while in turn selecting movements that make his own intentions legible and easy to infer by the other agent (i.e., sensorimotor communication). Over time, the interactive inference aligns both the beliefs and the behavioral strategies of the agents, hence ensuring the success of the joint action. We exemplify the functioning of the model in two simulations. The first simulation illustrates a ''leaderless'' joint action. It shows that when two agents lack a strong preference about their joint task goal, they jointly infer it by observing each other's movements. In turn, this helps the interactive alignment of their beliefs and behavioral strategies. The second simulation illustrates a "leader-follower" joint action. It shows that when one agent ("leader") knows the true joint goal, it uses sensorimotor communication to help the other agent ("follower") infer it, even if doing this requires selecting a more costly individual plan. These simulations illustrate that interactive inference supports successful multi-agent joint actions and reproduces key cognitive and behavioral dynamics of "leaderless" and "leader-follower" joint actions observed in human-human experiments. In sum, interactive inference provides a cognitively inspired, formal framework to realize cooperative joint actions and consensus in multi-agent systems.

Artificial Intelligence,Information Theory,Multiagent Systems,Neurons and Cognition
Active Inference and Reinforcement Learning: A unified inference on continuous state and action spaces under partial observability

Parvin Malekzadeh,Konstantinos N. Plataniotis

2024-06-01

Abstract:Reinforcement learning (RL) has garnered significant attention for developing decision-making agents that aim to maximize rewards, specified by an external supervisor, within fully observable environments. However, many real-world problems involve partial observations, formulated as partially observable Markov decision processes (POMDPs). Previous studies have tackled RL in POMDPs by either incorporating the memory of past actions and observations or by inferring the true state of the environment from observed data. However, aggregating observed data over time becomes impractical in continuous spaces. Moreover, inference-based RL approaches often require many samples to perform well, as they focus solely on reward maximization and neglect uncertainty in the inferred state. Active inference (AIF) is a framework formulated in POMDPs and directs agents to select actions by minimizing a function called expected free energy (EFE). This supplies reward-maximizing (exploitative) behaviour, as in RL, with information-seeking (exploratory) behaviour. Despite this exploratory behaviour of AIF, its usage is limited to discrete spaces due to the computational challenges associated with EFE. In this paper, we propose a unified principle that establishes a theoretical connection between AIF and RL, enabling seamless integration of these two approaches and overcoming their aforementioned limitations in continuous space POMDP settings. We substantiate our findings with theoretical analysis, providing novel perspectives for utilizing AIF in the design of artificial agents. Experimental results demonstrate the superior learning capabilities of our method in solving continuous space partially observable tasks. Notably, our approach harnesses information-seeking exploration, enabling it to effectively solve reward-free problems and rendering explicit task reward design by an external supervisor optional.

Machine Learning,Artificial Intelligence

Program of the Seventy-First Annual Meeting of the Potato Association of America at St. Paul Hotel St. Paul, Minnesota August 2–6, 1987

Deconstructing deep active inference

Deep Active Inference as Variational Policy Gradients

A Minimal Active Inference Agent

Active inference: demystified and compared

Active Inference as a Model of Agency

Inwardly rectifying K+ currents in intermediate cells in the cochlea of gerbils: a possible contribution to the endocochlear potential

Active inference on discrete state-spaces: a synthesis

Demonstrating the Continual Learning Capabilities and Practical Application of Discrete-Time Active Inference

Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop

On efficient computation in active inference

On Predictive planning and counterfactual learning in active inference

Active Inference on the Edge: A Design Study

Artificial enactive inference in three-dimensional world

Learning and embodied decisions in active inference

Active inference, Bayesian optimal design, and expected utility

Learning in Hybrid Active Inference Models

Interactive inference: a multi-agent model of cooperative joint actions

Active Inference and Reinforcement Learning: A unified inference on continuous state and action spaces under partial observability