Abstract:This paper introduces a model-based approach for training feedback controllers for an autonomous agent operating in a highly nonlinear (albeit deterministic) environment. We desire the trained policy to ensure that the agent satisfies specific task objectives and safety constraints, both expressed in Discrete-Time Signal Temporal Logic (DT-STL). One advantage for reformulation of a task via formal frameworks, like DT-STL, is that it permits quantitative satisfaction semantics. In other words, given a trajectory and a DT-STL formula, we can compute the {\em robustness}, which can be interpreted as an approximate signed distance between the trajectory and the set of trajectories satisfying the formula. We utilize feedback control, and we assume a feed forward neural network for learning the feedback controller. We show how this learning problem is similar to training recurrent neural networks (RNNs), where the number of recurrent units is proportional to the temporal horizon of the agent's task objectives. This poses a challenge: RNNs are susceptible to vanishing and exploding gradients, and naïve gradient descent-based strategies to solve long-horizon task objectives thus suffer from the same problems. To tackle this challenge, we introduce a novel gradient approximation algorithm based on the idea of dropout or gradient sampling. One of the main contributions is the notion of {\em controller network dropout}, where we approximate the NN controller in several time-steps in the task horizon by the control input obtained using the controller in a previous training step. We show that our control synthesis methodology, can be quite helpful for stochastic gradient descent to converge with less numerical issues, enabling scalable backpropagation over long time horizons and trajectories over high dimensional state spaces.

Counter-example guided Imitation Learning of Feedback Controllers from Temporal Logic Specifications

Signal Temporal Logic Neural Predictive Control

Learning Robust and Correct Controllers from Signal Temporal Logic Specifications Using BarrierNet

Model-based Reinforcement Learning from Signal Temporal Logic Specifications

Deep Reinforcement Learning Based Networked Control with Network Delays for Signal Temporal Logic Specifications

Safe Model-based Control from Signal Temporal Logic Specifications Using Recurrent Neural Networks

A Neurosymbolic Approach to the Verification of Temporal Logic Properties of Learning enabled Control Systems

Interpretable Generative Adversarial Imitation Learning

Continuous-time Control Synthesis Under Nested Signal Temporal Logic Specifications

Scaling Learning based Policy Optimization for Temporal Logic Tasks by Controller Network Dropout

Tractable Reinforcement Learning of Signal Temporal Logic Objectives

Funnel-based Reward Shaping for Signal Temporal Logic Tasks in Reinforcement Learning

Controller Synthesis of Collaborative Signal Temporal Logic Tasks for Multi-Agent Systems via Assume-Guarantee Contracts

Direct Learning Control of Trajectories Subject to High-Order Internal Model for a Class of Continuous-Time Linear Systems.

Signal Temporal Logic Control Synthesis among Uncontrollable Dynamic Agents with Conformal Prediction

Control from Signal Temporal Logic Specifications with Smooth Cumulative Quantitative Semantics

Direct data-driven control with signal temporal logic specifications

Verified Compositions of Neural Network Controllers for Temporal Logic Control Objectives

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation

Tunable Reactive Synthesis for Lipschitz-Bounded Systems with Temporal Logic Specifications