What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve end - to - end control of the Formula Student (FS) racing car in autonomous driving using Deep Reinforcement Learning (RL). Specifically, the paper explores training two state - of - the - art RL algorithms (TD3 and DQN) in a simulated environment and testing the performance of these algorithms on an autonomous FS racing car in the real world to verify their applicability and performance on the actual track. In this way, the research aims to overcome the limitations of traditional path - planning methods and improve the adaptability of robots in dynamic and unpredictable environments, especially for the autonomous racing car scenario. ### Background of the Paper and Related Work - **Reinforcement Learning (RL)**: As a branch of artificial intelligence, RL learns appropriate control strategies through direct interaction with the environment and can adapt to new situations without human - designed solutions. This method shows great potential in robot control problems. - **Mobile Robot Control**: Although traditional methods have been successful in solving the motion planning and control problems of mobile robots, they usually require a great deal of engineering effort to be reliably deployed in the real world. Machine - learning - based navigation methods, especially Deep Reinforcement Learning (DRL), have been proposed as a way to reduce this manual engineering effort. - **Challenges**: The main challenges faced by DRL methods include performance verification, narrowing the gap between simulation and reality, sample efficiency, designing practical reward functions, and ensuring safety. ### Methods - **Experimental Setup**: The Turtlebot2 platform equipped with a Realsense D435 camera is used for simulation and real - world training. To simplify the computer vision challenges, the conical signs on the track are replaced with ArUco markers. - **State Space and Action Space**: - **State Space**: Consists of the positions of the six nearest ArUco markers detected by the RealSense D435 camera. The position information includes the lateral distance (x) and the forward distance (z) of the markers relative to the Turtlebot2. - **Action Space**: The action space of the DQN model is discrete positive and negative fixed rotation speeds (±0.2 rad/s), while the action space of the TD3 model is continuous rotation speeds (- 0.4 rad/s to 0.4 rad/s). - **Reward Function**: Based on the angle difference between the current direction of the robot and the direction of the midpoint of the nearest pair of markers, a cosine function is used to define the reward to encourage the robot to stay on the center line of the track. ### Results - **Simulation Training Results**: The TD3 model begins to stabilize after 2000 training cycles, while the average reward of the DQN model is still increasing after 5000 cycles. - **Simulation Track Segment Tests**: The TD3 model shows higher success rates and completion degrees on various combinations of track segments. - **Real - World Track Segment Tests**: The TD3 model performs better than the DQN model in the real world, especially when turning left. - **Oval Track Simulation**: The TD3 model also performs more stably on the oval track, but all models have difficulties in completing the entire track, mainly having problems when turning. ### Discussion - **Rotation Control Jitter**: The models show high jitter when turning, which may be because the reward function is based only on the angle rather than the rate of change of the action. Introducing a complex reward function that evaluates the smoothness of rotation control may significantly reduce the jitter. - **Model Evaluation**: The TD3 model outperforms the DQN model in all tests, especially the TD3 model trained without noise performs the best. Adding noise has little impact on performance, and in some cases, the no - noise model performs even better. ### Conclusion This research shows the preliminary results of using deep reinforcement learning to achieve end - to - end control of autonomous FS racing cars, but further improvement is still needed, especially in reducing rotation control jitter and improving performance on complex tracks. Future research can explore more complex reward functions and more training data to further optimize the model performance.

Racing Towards Reinforcement Learning based control of an Autonomous Formula SAE Car

Deep Reinforcement Learning for Local Path Following of an Autonomous Formula SAE Vehicle

Formula RL: Deep Reinforcement Learning for Autonomous Racing using Telemetry Data

DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning

Towards Safety Assured End-to-End Vision-Based Control for Autonomous Racing

Self-Driving Car Racing: Application of Deep Reinforcement Learning

FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing

A LiDAR-based approach to autonomous racing with model-free reinforcement learning

DeepRacing: Parameterized Trajectories for Autonomous Racing

Vehicle Extreme Control Based on Offline Reinforcement Leaning

Vision-based control in the open racing car simulator with deep and reinforcement learning

Autonomous Racing using a Hybrid Imitation-Reinforcement Learning Architecture

Explorations and Lessons Learned in Building an Autonomous Formula SAE Car from Simulations

Learning Autonomous Race Driving with Action Mapping Reinforcement Learning

Autonomous Formula Racecar: Overall System Design and Experimental Validation

Towards Optimal Head-to-head Autonomous Racing with Curriculum Reinforcement Learning

Constrained Residual Race: an Efficient Hybrid Controller for Autonomous Racing

Learn-to-Race: A Multimodal Control Environment for Autonomous Racing

Learn 2 Rage: Experiencing The Emotional Roller Coaster That Is Reinforcement Learning

High-speed Autonomous Racing using Trajectory-aided Deep Reinforcement Learning

From perception to control: an autonomous driving system for a formula student driverless car