Abstract:Standard approaches to controlling dynamical systems involve biologically implausible steps such as backpropagation of errors or intermediate model-based system representations. Recent advances in machine learning have shown that "imperfect" feedback of errors during training can yield test performance that is similar to using full backpropagated errors, provided that the two error signals are at least somewhat aligned. Inspired by such methods, we introduce an iterative, spatiotemporally local protocol to learn driving forces and control non-equilibrium dynamical systems using imperfect feedback signals. We present numerical experiments and theoretical justification for several examples. For systems in conservative force fields that are driven by external time-dependent protocols, our update rules resemble a dynamical version of contrastive divergence. We appeal to linear response theory to establish that our imperfect update rules are locally convergent for these conservative systems. For systems evolving under non-conservative dynamics, we derive a new theoretical result that makes possible the control of non-equilibrium steady-state probabilities through simple local update rules. Finally, we show that similar local update rules can also solve dynamical control problems for non-conservative systems, and we illustrate this in the non-trivial example of active nematics. Our updates allow learning spatiotemporal activity fields that pull topological defects along desired trajectories in the active nematic fluid. These imperfect feedback methods are information efficient and in principle biologically plausible, and they can help extend recent methods of decentralized training for physical materials into dynamical settings.

Pathwise Relaxed Optimal Control of Rough Differential Equations

Pathwise Stochastic Control and a Class of Stochastic Partial Differential Equations

Pathwise Optimal Control and Rough Fractional Hamilton-Jacobi-Bellman Equations for Rough-Fractional Dynamics

On the stability of Lipschitz continuous control problems and its application to reinforcement learning

Regularity and stability of feedback relaxed controls

Optimal Control of Unbounded Functional Stochastic Evolution Systems in Hilbert Spaces: Second-Order Path-dependent HJB Equation

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Optimal control of a non-smooth elliptic PDE with non-linear term acting on the control

Relaxed optimal control for the stochastic Landau-Lifshitz-Bloch equation with jump noise

Relaxed formulation for Controlled Branching Diffusions, Existence of an Optimal Control and HJB Equation

Optimal control problems with generalized mean-field dynamics and viscosity solution to Master Bellman equation

Leveraging Randomized Smoothing for Optimal Control of Nonsmooth Dynamical Systems

Robustness of Optimal Controlled Diffusions with Near-Brownian Noise via Rough Paths Theory

Relaxation and asymptotic expansion of controlled stiff differential equations

Rough Differential Equations Containing Path-Dependent Bounded Variation Terms

Existence and optimality conditions for relaxed mean-field stochastic control problems

Optimal control of infinite-dimensional differential systems with randomness and path-dependence and stochastic path-dependent Hamilton-Jacobi equations

Stochastic Near-Optimal Controls for Path-Dependent Systems

Second Order Optimality Conditions for Optimal Control Problems of Stochastic Evolution Equations

Learning to control non-equilibrium dynamics using local imperfect gradients

Robust feedback control of nonlinear PDEs by numerical approximation of high-dimensional Hamilton-Jacobi-Isaacs equations