Abstract:Reinforcement learning (RL) has demonstrated considerable potential in addressing intricate control and decision problems such as vehicle tracking control and obstacle avoidance. Nonetheless, the control policies acquired through RL often lack smoothness even in the presence of minor noises or disturbances, which may induce oscillations, overheating, and even damage to the system in real-world applications. Existing methods handle this issue from temporal or spatial domains and are often tightly coupled with specific tasks. This paper studies the smoothness of control policy from the frequency domain perspective. We propose a class of neural networks with low-pass filtering ability, named Smonet, to alleviate the non-smooth issue by learning a low-frequency representation within hidden layers. Smonet features with serial filtering layers responsible for low-pass filtering of the input signal. Each filtering layer contains multiple inertia cells, one adaptive cell, and one activation layer. To facilitate the filtering ability of Smonet, we further proposed a Smonet-based RL training method by integrating an extra regularization term relating to filtering factors to standard RL loss. Finally, we assess the efficacy of Smonet through diverse simulated robot control tasks and a real-world mobile robot obstacle avoidance experiment, comparing its performance with two commonly utilized networks, multi-layer perceptron and gated recurrent unit. Results indicate that Smonet consistently enhances policy smoothness under various observation noises without compromising control performance. Notably, it achieves up to a 72.7% reduction in the action fluctuation ratio compared to traditional network structures.

RTSNet: Deep Learning Aided Kalman Smoothing

RTSNet: Deep Learning Aided Kalman Smoothing

RTSNet: Learning to Smooth in Partially Known State-Space Models (Preprint)

A nonlinear smoother for target tracking in asynchronous wireless sensor networks

KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics

Performance Evaluation of Deep Learning-Based State Estimation: A Comparative Study of KalmanNet

Generalized Kalman Smoothing: Modeling and Algorithms

Latent-KalmanNet: Learned Kalman Filtering for Tracking from High-Dimensional Signals

Deep Neural Network-Embedded Stochastic Nonlinear State-Space Models and Their Applications to Process Monitoring

Robust and Trend Following Student's t Kalman Smoothers

RKFNet: A Novel Neural Network Aided Robust Kalman Filter

Backward smoothing for precise GNSS applications

GSP-KalmanNet: Tracking Graph Signals via Neural-Aided Kalman Filtering

Switching Recurrent Kalman Networks

Accurate Smoothing Methods for State Estimation of Continuous-Discrete Nonlinear Dynamic Systems

Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning.

Adaptive KalmanNet: Data-Driven Kalman Filter with Fast Adaptation

Transformer-Based Multi-Object Smoothing with Decoupled Data Association and Smoothing

Kalman smoothing and block tridiagonal systems: new connections and numerical stability results

Smooth Filtering Neural Network for Reinforcement Learning