Abstract:In this paper, we present a Deep Reinforcement Learning (RL)-driven Adaptive Stochastic Nonlinear Model Predictive Control (SNMPC) to optimize uncertainty handling, constraints robustification, feasibility, and closed-loop performance. To this end, we conceive an RL agent to proactively anticipate upcoming control tasks and to dynamically determine the most suitable combination of key SNMPC parameters - foremost the robustification factor $\kappa$ and the Uncertainty Propagation Horizon (UPH) $T_u$. We analyze the trained RL agent's decision-making process and highlight its ability to learn context-dependent optimal parameters. One key finding is that adapting the constraints robustification factor with the learned policy reduces conservatism and improves closed-loop performance while adapting UPH renders previously infeasible SNMPC problems feasible when faced with severe disturbances. We showcase the enhanced robustness and feasibility of our Adaptive SNMPC (aSNMPC) through the real-time motion control task of an autonomous passenger vehicle to follow an optimal race line when confronted with significant time-variant disturbances. Experimental findings demonstrate that our look-ahead RL-driven aSNMPC outperforms its Static SNMPC (sSNMPC) counterpart in minimizing the lateral deviation both with accurate and inaccurate disturbance assumptions and even when driving in previously unexplored environments.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to optimize uncertainty handling, constraint robustness, feasibility, and closed-loop performance in motion control of autonomous vehicles through Adaptive Stochastic Nonlinear Model Predictive Control (aSNMPC) combined with Look-ahead Deep Reinforcement Learning (DRL). Specifically, the paper proposes a DRL-based aSNMPC framework that can dynamically adjust key SNMPC parameters, such as the constraint robustness factor κ and the Uncertainty Propagation Horizon (UPH), to cope with uncertainties in different environments and dynamic situations, thereby improving the robustness and performance of the control system. ### Main Contributions: 1. **Proposed an adaptive SNMPC framework based on look-ahead DRL**: This framework can automatically adjust SNMPC parameters based on current closed-loop performance evaluation and future control reference trajectories, thereby better handling various uncertainties and different dynamic situations. 2. **Demonstrated improvements in robustness and feasibility of aSNMPC compared to standard SNMPC**: Especially when facing significant time-varying disturbances, aSNMPC shows better performance and higher feasibility. 3. **Analyzed the impact of each parameter on SNMPC performance**: Through experiments, the specific impact of learning and adjusting each parameter on system performance was verified, and a context-dependent decision process analysis was conducted. ### Specific Problems Addressed: - **Uncertainty Handling**: By dynamically adjusting κ and UPH, the system's uncertainty is better managed, reducing conservativeness and improving control accuracy. - **Constraint Robustness**: Ensuring that the control system can meet state and control constraints in the face of uncertainties and disturbances, avoiding frequent constraint violations. - **Feasibility**: By adjusting UPH, preventing infeasibility issues caused by uncertainty propagation, ensuring feasible control strategies can be found in various dynamic situations. - **Closed-loop Performance**: Optimizing closed-loop performance, reducing lateral deviation, and improving tracking accuracy and stability of the vehicle in complex environments. ### Experimental Validation: - **Performance Comparison on the Montblanc Track**: Under significant time-varying disturbances, aSNMPC significantly outperforms standard SNMPC in terms of lateral deviation. - **Generalization Ability on Different Tracks**: On unseen Modena and Las Vegas tracks, aSNMPC also shows excellent performance and robustness, especially when facing inaccurate disturbance assumptions. ### Conclusion and Future Work: - **Conclusion**: The proposed aSNMPC framework has achieved significant results in optimizing uncertainty handling, constraint robustness, feasibility, and closed-loop performance, especially in complex dynamic environments. - **Future Work**: Further research on how to apply this method to more types of autonomous driving tasks, explore more efficient DRL algorithms, and conduct experimental validation on actual vehicles. Through these contributions, the paper provides a new effective solution for motion control of autonomous vehicles, improving the robustness and performance of the system.

Adaptive Stochastic Nonlinear Model Predictive Control with Look-ahead Deep Reinforcement Learning for Autonomous Vehicle Motion Control

Model-Based Robot Learning Control with Uncertainty Directed Exploration

A Stochastic Nonlinear Model Predictive Control with an Uncertainty Propagation Horizon for Autonomous Vehicle Motion Control

Adaptive robust control algorithm for enhanced path-tracking performance of automated driving in critical scenarios

Learning-Based Neural Dynamic Surface Predictive Control for MMC

A Learning-Based Nonlinear Model Predictive Control Approach for Autonomous Driving

R$^2$NMPC: A Real-Time Reduced Robustified Nonlinear Model Predictive Control with Ellipsoidal Uncertainty Sets for Autonomous Vehicle Motion Control

Uncertainty-aware hybrid paradigm of nonlinear MPC and model-based RL for offroad navigation: Exploration of transformers in the predictive model

Robust Learning Control for Autonomous Vehicle with Network Delays and Disturbances

Adaptive Robust Model Predictive Control via Uncertainty Cancellation

Adaptive Fault Tolerant Control for Safe Autonomous Driving Using Learning-based Model Predictive Control

Predictive Control for Autonomous Driving with Uncertain, Multi-modal Predictions

Incorporating Recurrent Reinforcement Learning into Model Predictive Control for Adaptive Control in Autonomous Driving

Friction-Adaptive Stochastic Nonlinear Model Predictive Control for Autonomous Vehicles

Robust Car-Following Control of Connected and Autonomous Vehicles: A Stochastic Model Predictive Control Approach

Adaptive Nonlinear Model Predictive Horizon Using Deep Reinforcement Learning for Optimal Trajectory Planning

Optimal reinforcement learning and probabilistic-risk-based path planning and following of autonomous vehicles with obstacle avoidance

A Safe Reinforcement Learning driven Weights-varying Model Predictive Control for Autonomous Vehicle Motion Control

Synthesis of Robust Model Predictive Control for Vehicle Path Following with Uncertainties

Performance-Guaranteed Adaptive Optimized Control of Intelligent Surface Vehicle Using Reinforcement Learning

Nonlinear Model Predictive Control of Autonomous Vehicles Considering Dynamic Stability Constraints