[Calcification pattern of rat and rabbit incisor dentin].

H. Mishima,T. Terajima,K. Matsushita,Y. Kozawa

1987-09-01

Abstract:

What problem does this paper attempt to address?

Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challenges

Nan Cheng,Xiucheng Wang,Zan Li,Zhisheng Yin,Tom Luan,Xuemin Shen

2024-06-16

Abstract:This article presents a digital twin (DT)-enhanced reinforcement learning (RL) framework aimed at optimizing performance and reliability in network resource management, since the traditional RL methods face several unified challenges when applied to physical networks, including limited exploration efficiency, slow convergence, poor long-term performance, and safety concerns during the exploration phase. To deal with the above challenges, a comprehensive DT-based framework is proposed to enhance the convergence speed and performance for unified RL-based resource management. The proposed framework provides safe action exploration, more accurate estimates of long-term returns, faster training convergence, higher convergence performance, and real-time adaptation to varying network conditions. Then, two case studies on ultra-reliable and low-latency communication (URLLC) services and multiple unmanned aerial vehicles (UAV) network are presented, demonstrating improvements of the proposed framework in performance, convergence speed, and training cost reduction both on traditional RL and neural network based Deep RL (DRL). Finally, the article identifies and explores some of the research challenges and open issues in this rapidly evolving field.

Systems and Control,Machine Learning,Networking and Internet Architecture
Dense reinforcement learning for safety validation of autonomous vehicles

Shuo Feng,Haowei Sun,Xintao Yan,Haojie Zhu,Zhengxia Zou,Shengyin Shen,Henry X. Liu

DOI: https://doi.org/10.1038/s41586-023-05732-2

IF: 64.8

2023-03-23

Nature

Abstract:One critical bottleneck that impedes the development and deployment of autonomous vehicles is the prohibitively high economic and time costs required to validate their safety in a naturalistic driving environment, owing to the rarity of safety-critical events 1 . Here we report the development of an intelligent testing environment, where artificial-intelligence-based background agents are trained to validate the safety performances of autonomous vehicles in an accelerated mode, without loss of unbiasedness. From naturalistic driving data, the background agents learn what adversarial manoeuvre to execute through a dense deep-reinforcement-learning (D2RL) approach, in which Markov decision processes are edited by removing non-safety-critical states and reconnecting critical ones so that the information in the training data is densified. D2RL enables neural networks to learn from densified information with safety-critical events and achieves tasks that are intractable for traditional deep-reinforcement-learning approaches. We demonstrate the effectiveness of our approach by testing a highly automated vehicle in both highway and urban test tracks with an augmented-reality environment, combining simulated background vehicles with physical road infrastructure and a real autonomous test vehicle. Our results show that the D2RL-trained agents can accelerate the evaluation process by multiple orders of magnitude (10 3 to 10 5 times faster). In addition, D2RL will enable accelerated testing and training with other safety-critical autonomous systems.

multidisciplinary sciences
Deep Reinforcement Learning for Online Resource Allocation in IoT Networks: Technology, Development, and Future Challenges

Peng Cheng,Youjia Chen,Ming Ding,Zhuo Chen,Sige Liu,Yi-Ping Phoebe Chen

DOI: https://doi.org/10.1109/mcom.001.2200526

IF: 9.03

2023-06-20

IEEE Communications Magazine

Abstract:The growing number of complex and heterogeneous Internet of Things (IoT) applications has imposed a high demand for scarce communications and computing resources. To meet this stringent requirement, it is desirable to develop large-scale highly adaptive online resource allocation strategies to streamline existing network operations. Deep reinforcement learning (DRL), which combines the merits of reinforcement learning and deep learning, is capable of addressing complex decision-making tasks, thus enabling efficient online resource allocation. In this article, we present a DRL-based resource allocation framework. We begin a discussion on DRL basics and review its several recent applications. Then, we develop two new DRL algorithms that facilitate unlocking the potential of DRL and offer viable solutions to many more complex resource allocation problems. The first one tackles an optimization problem exposed to mixed (discrete and continuous) action spaces and bound by a number of highly non-linear quality-of-service (QoS) constraints. The second one extends the single-agent DRL to a more challenging multi-agent DRL by introducing a novel semi-distributed architecture. Finally, we discuss the challenges and future visions of applying DRL to real-world IoT networks.

telecommunications,engineering, electrical & electronic
A Digital Twin Framework for Reinforcement Learning with Real-Time Self-Improvement via Human Assistive Teleoperation

Kabirat Olayemi,Mien Van,Luke Maguire,Sean McLoone

2024-06-02

Abstract:Reinforcement Learning (RL) or Deep Reinforcement Learning (DRL) is a powerful approach to solving Markov Decision Processes (MDPs) when the model of the environment is not known a priori. However, RL models are still faced with challenges such as handling covariate shifts and ensuring the quality of human demonstration. To address these challenges and further advance DRL models, our work develops a human-in-the-loop DRL framework via digital twin that leverages human intelligence after deployment to retrain the DRL model in real time. First, we develop a pre-trained model fully based on learning through trial and error in the simulated environment allowing scalability and automation while eliminating variability and biases that can come from subjective human guidance. Second, instead of deploying the trained model directly on the UGV, we create a digital twin which controls the physical UGV from the virtual environment. Third, to allow continuous learning without catastrophic forgetting, we introduce the ability of the model to self-improve with the help of small human guidance at the start of the retraining. We test the performance of our proposed model in both simulation and real-world environments with both static and dynamic obstacles. The results indicate that our proposed approach not only outperforms the baseline models in terms of reward accumulation but also demonstrates superior training efficiency.

Systems and Control
Digital twin-based reinforcement learning framework: application to autonomous mobile robot dispatching

Amel Jaoua,Samar Masmoudi,Elisa Negri

DOI: https://doi.org/10.1080/0951192x.2024.2314787

IF: 4.1

2024-02-16

International Journal of Computer Integrated Manufacturing

Abstract:This paper proposes a new framework for embedding an Intelligent Digital Twin (DT) in a production system with the objective of achieving more efficient real-time production planning and control. For that purpose, the Intelligence Layer is based on Reinforcement Leaning (RL) and Deep RL (DRL) algorithms. The use of this control instead of parametric simulation-based optimization approach allows to benefit from the separation between the training and execution phase. To ensure consistency and reusability, this work presents a standardized framework, based on a formal methodology, that specifies how the various components of the DT-based RL architecture interact over time to achieve essential real-time concurrency and synchronization aspects. Experiments are conducted in a small-scale production system where material handling operations are performed by an Autonomous Mobile Robot (AMR) in an Industry 4.0 Laboratory. Results showed how synchronized state updates between the Physical and Cyber World are used within the Decision Layer to ensure real-time response for the AMR dispatching requests. Finally, to deal with continuous and high-dimensional state spaces, the Deep Q-Network is implemented. The findings of an extensive computational study reveal that implementing the DT-based DRL solution leads to improved efficiency and robustness when compared to conventional dispatching rules.

engineering, manufacturing,computer science, interdisciplinary applications,operations research & management science
Deep Reinforcement Learning (DRL)-based Resource Management in Software-Defined and Virtualized Vehicular Ad Hoc Networks

Ying He,F. Richard Yu,Nan Zhao,Hongxi Yin,Azzedine Boukerche

DOI: https://doi.org/10.1145/3132340.3132355

2017-01-01

Abstract:Vehicular ad hoc networks (VANETs) have attracted great interests from both industry and academia. The developments of VANETs are heavily influenced by information and communications technologies, which have fueled a plethora of innovations in various areas, including networking, caching and computing. Nevertheless, these important enabling technologies have traditionally been studied separately in the existing works on vehicular networks. In this paper, we propose an integrated framework that can enable dynamic orchestration of networking, caching and computing resources to improve the performance of next generation vehicular networks. We formulate the resource allocation strategy in this framework as a joint optimization problem, where the gains of not only networking but also caching and computing are taken into consideration in the proposed framework. The complexity of the system is very high when we jointly consider these three technologies. Therefore, we propose a novel deep reinforcement learning approach in this paper. Simulation results with different system parameters are presented to show the effectiveness of the proposed scheme.
Resource Constrained Deep Reinforcement Learning

Abhinav Bhatia,Pradeep Varakantham,Akshat Kumar

DOI: https://doi.org/10.48550/arXiv.1812.00600

IF: 5.414

2018-12-03

Machine Learning

Abstract:In urban environments, supply resources have to be constantly matched to the "right" locations (where customer demand is present) so as to improve quality of life. For instance, ambulances have to be matched to base stations regularly so as to reduce response time for emergency incidents in EMS (Emergency Management Systems); vehicles (cars, bikes, scooters etc.) have to be matched to docking stations so as to reduce lost demand in shared mobility systems. Such problem domains are challenging owing to the demand uncertainty, combinatorial action spaces (due to allocation) and constraints on allocation of resources (e.g., total resources, minimum and maximum number of resources at locations and regions). Existing systems typically employ myopic and greedy optimization approaches to optimize allocation of supply resources to locations. Such approaches typically are unable to handle surges or variances in demand patterns well. Recent research has demonstrated the ability of Deep RL methods in adapting well to highly uncertain environments. However, existing Deep RL methods are unable to handle combinatorial action spaces and constraints on allocation of resources. To that end, we have developed three approaches on top of the well known actor critic approach, DDPG (Deep Deterministic Policy Gradient) that are able to handle constraints on resource allocation. More importantly, we demonstrate that they are able to outperform leading approaches on simulators validated on semi-real and real data sets.
Joint Optimization of Sensing, Decision-making and Motion-controlling for Autonomous Vehicles: A Deep Reinforcement Learning Approach

Longquan Chen,Ying He,Qiang Wang,Weike Pan,Zhong Ming

DOI: https://doi.org/10.1109/tvt.2022.3150793

IF: 6.8

2022-01-01

IEEE Transactions on Vehicular Technology

Abstract:The three main modules of autonomous vehicles, i.e., sensing, decision making, and motion controlling, have been studied separately in most existing works on autonomous driving, which overlook the correlations among these modules, leading to a result of unsatisfactory performance. In this paper, we propose a novel scheme that first tactfully processes the sensing data, then jointly learns and optimizes the decision-making and motion-controlling using reinforcement learning (RL). Specifically, the proposed scheme designs a novel state representation mechanism, where the sensing data goes through the attention layer and the convolutional neural network (CNN) layer sequentially. The attention layer focuses on extracting the most important local information and then CNN layer takes a broad view to comprehensively consider the global information for a better representation. Furthermore, the proposed scheme jointly learns decision-making and motion-controlling, therefore, the relevance of these two modules is implicitly considered, which helps to achieve a better autonomous driving policy. Extensive simulation results show that the proposed scheme is better than classic control methods and some RL methods in terms of safety, velocity, etc. We also demonstrate the respective functions of the attention layer and the CNN layer through ablation studies. Finally, we construct a traffic scene with a real autonomous vehicle, and verified the feasibility of the proposed scheme.

telecommunications,engineering, electrical & electronic,transportation science & technology
From Naturalistic Traffic Data to Learning-Based Driving Policy: A Sim-to-Real Study

Kevin Mi,Mingfeng Yuan,J. Shan

DOI: https://doi.org/10.1109/TVT.2023.3307409

IF: 6.8

2024-01-01

IEEE Transactions on Vehicular Technology

Abstract:Reinforcement learning (RL) is a promising way to achieve human-like autonomous driving (HAD) in complex and dynamic traffic, but faces challenges such as low sample efficiency, partial observability, and sim2real transfer. In light of this, a comprehensive solution for RL-driven HAD is established. First, an efficient training scheme called Deep Recurrent Q-learning from demonstration algorithm (DRQfD) is proposed for lane-changing decision-making to address the low sample efficiency in RL and the poor generalization capability in Imitation Learning (IL). The inherent LSTM structure potentially learns to predict future states of surrounding vehicles, helping to address the partially observable problem in autonomous driving (AD). Second, to reduce the sim2real gap, a twin high-fidelity simulator is built based on ROS-Gazebo for simulating LiDAR sensing, model training, and evaluations. Domain randomization is used to improve the robustness and generalization ability, making it easier for the model to be transferred to real-world scenarios. In addition, for the multi-objective optimization and imbalanced data issues in this scenario, a hierarchical decision-making framework is proposed to decompose the complex decision-making problem into several subtasks, making the driving policies easier to converge. To avoid the excessive dependence of the decision-making module on the output of perception module in modular systems, we train each modularized skill in an end-to-end manner. Moreover, we compare our method with a vanilla RL method to show improvement in learning efficiency. Comparisons between RL-based model and IL baseline in terms of safety, travel efficiency, and human-likeness are also given. To further validate the generalization ability of our model, we test the model on real traffic dataset. Finally, we implement the RL model on physical cars to demonstrate the performance of sim2real transfer.

Computer Science,Engineering
Deep Reinforcement Learning Enables Joint Trajectory and Communication in Internet of Robotic Things

Ruyu Luo,Hui Tian,Wanli Ni,Julian Cheng,Kwang-Cheng Chen

DOI: https://doi.org/10.1109/twc.2024.3462450

IF: 10.4

2024-01-01

IEEE Transactions on Wireless Communications

Abstract:Internet of Robotic Things (IoRT) emphasizes the integrated robotic, artificial intelligence computing, and communication technologies, enabling more sophisticated operations and decision-making. As a crucial element of IoRT, mission-critical applications, such as industrial manufacturing and emergency services, impose stringent requirements on ultra-reliable and low-latency communication (URLLC). The paper focuses on addressing URLLC challenges in the context of IoRT, particularly when autonomous mobile robots (AMRs) coexist with static sensors. We prioritize safe and efficient AMRs’ travel through trajectory design and communication resource allocation in IoRT systems without the need of any prior knowledge. To enhance network connectivity and exploit diversity gains, we introduce the flexible decoding and free clustering as the next-generation multiple access technologies in spectrum-limited downlink IoRT system. Then, aiming at minimizing the decoding error probability and travel time, we formulate a long-term multi-objective optimization problem by jointly designing AMRs’ trajectory and communication resource. To accommodate the inherent dynamics and unpredictability in the IoRT system, we introduce a multi-agent actor-critic deep reinforcement learning (DRL) framework, offering four distinct implementations, each accompanied by comprehensive complexity analyses. Simulation results reveal the following insights: 1) in terms of DRL implementations, off-policy algorithms with deterministic policies outperform their on-policy counterparts, achieving approximately a 67% increase in rewards; 2) In terms of communication schemes, our proposed flexible decoding and free clustering strategies under designed trajectories can effectively reduce decoding errors; 3) In terms of algorithm optimality, our DRL framework shows superior flexibility and adaptability in communication environments compared to traditional A* search and heuristic methods.
Data-Driven Distributed Common Operational Picture from Heterogeneous Platforms using Multi-Agent Reinforcement Learning

Indranil Sur,Aswin Raghavan,Abrar Rahman,James Z Hare,Daniel Cassenti,Carl Busart

2024-11-09

Abstract:The integration of unmanned platforms equipped with advanced sensors promises to enhance situational awareness and mitigate the "fog of war" in military operations. However, managing the vast influx of data from these platforms poses a significant challenge for Command and Control (C2) systems. This study presents a novel multi-agent learning framework to address this challenge. Our method enables autonomous and secure communication between agents and humans, which in turn enables real-time formation of an interpretable Common Operational Picture (COP). Each agent encodes its perceptions and actions into compact vectors, which are then transmitted, received and decoded to form a COP encompassing the current state of all agents (friendly and enemy) on the battlefield. Using Deep Reinforcement Learning (DRL), we jointly train COP models and agent's action selection policies. We demonstrate resilience to degraded conditions such as denied GPS and disrupted communications. Experimental validation is performed in the Starcraft-2 simulation environment to evaluate the precision of the COPs and robustness of policies. We report less than 5% error in COPs and policies resilient to various adversarial conditions. In summary, our contributions include a method for autonomous COP formation, increased resilience through distributed prediction, and joint training of COP models and multi-agent RL policies. This research advances adaptive and resilient C2, facilitating effective control of heterogeneous unmanned platforms.

Multiagent Systems,Artificial Intelligence
Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Changquan Wang,Yun Wang

DOI: https://doi.org/10.3390/s24103139

IF: 3.9

2024-05-16

Sensors

Abstract:Autonomous driving has the potential to revolutionize transportation, but developing safe and reliable systems remains a significant challenge. Reinforcement learning (RL) has emerged as a promising approach for learning optimal control policies in complex driving environments. However, existing RL-based methods often suffer from low sample efficiency and lack explicit safety constraints, leading to unsafe behaviors. In this paper, we propose a novel framework for safe reinforcement learning in autonomous driving that addresses these limitations. Our approach incorporates a latent dynamic model that learns the underlying dynamics of the environment from bird's-eye view images, enabling efficient learning and reducing the risk of safety violations by generating synthetic data. Furthermore, we introduce state-wise safety constraints through a barrier function, ensuring safety at each state by encoding constraints directly into the learning process. Experimental results in the CARLA simulator demonstrate that our framework significantly outperforms baseline methods in terms of both driving performance and safety. Our work advances the development of safe and efficient autonomous driving systems by leveraging the power of reinforcement learning with explicit safety considerations.

engineering, electrical & electronic,instruments & instrumentation,chemistry, analytical
Energy Conservation for Internet of Things Tracking Applications Using Deep Reinforcement Learning

Salman Md Sultan,Muhammad Waleed,Jae-Young Pyun,Tai-Won Um

DOI: https://doi.org/10.3390/s21093261

IF: 3.9

2021-05-08

Sensors

Abstract:The Internet of Things (IoT)-based target tracking system is required for applications such as smart farm, smart factory, and smart city where many sensor devices are jointly connected to collect the moving target positions. Each sensor device continuously runs on battery-operated power, consuming energy while perceiving target information in a particular environment. To reduce sensor device energy consumption in real-time IoT tracking applications, many traditional methods such as clustering, information-driven, and other approaches have previously been utilized to select the best sensor. However, applying machine learning methods, particularly deep reinforcement learning (Deep RL), to address the problem of sensor selection in tracking applications is quite demanding because of the limited sensor node battery lifetime. In this study, we proposed a long short-term memory deep Q-network (DQN)-based Deep RL target tracking model to overcome the problem of energy consumption in IoT target applications. The proposed method is utilized to select the energy-efficient best sensor while tracking the target. The best sensor is defined by the minimum distance function (i.e., derived as the state), which leads to lower energy consumption. The simulation results show favorable features in terms of the best sensor selection and energy consumption.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Reinforcement Learning from Simulation to Real World Autonomous Driving using Digital Twin

Kevin Voogd,Jean Pierre Allamaa,Javier Alonso-Mora,Tong Duy Son

DOI: https://doi.org/10.48550/arXiv.2211.14874

2022-11-28

Abstract:Reinforcement learning (RL) is a promising solution for autonomous vehicles to deal with complex and uncertain traffic environments. The RL training process is however expensive, unsafe, and time consuming. Algorithms are often developed first in simulation and then transferred to the real world, leading to a common sim2real challenge that performance decreases when the domain changes. In this paper, we propose a transfer learning process to minimize the gap by exploiting digital twin technology, relying on a systematic and simultaneous combination of virtual and real world data coming from vehicle dynamics and traffic scenarios. The model and testing environment are evolved from model, hardware to vehicle in the loop and proving ground testing stages, similar to standard development cycle in automotive industry. In particular, we also integrate other transfer learning techniques such as domain randomization and adaptation in each stage. The simulation and real data are gradually incorporated to accelerate and make the transfer learning process more robust. The proposed RL methodology is applied to develop a path following steering controller for an autonomous electric vehicle. After learning and deploying the real-time RL control policy on the vehicle, we obtained satisfactory and safe control performance already from the first deployment, demonstrating the advantages of the proposed digital twin based learning process.

Robotics
Broad Reinforcement Learning for Supporting Fast Autonomous IoT

Xin Wei,Jialin Zhao,Liang Zhou,Yi Qian

DOI: https://doi.org/10.1109/jiot.2020.2980198

IF: 10.6

2020-08-01

IEEE Internet of Things Journal

Abstract:The emergence of a massive Internet-of-Things (IoT) ecosystem is changing the human lifestyle. In several practical scenarios, IoT still faces significant challenges with reliance on human assistance and unacceptable response time for the treatment of big data. Therefore, it is very urgent to establish a new framework and algorithm to solve problems specific to this kind of fast autonomous IoT. Traditional reinforcement learning and deep reinforcement learning (DRL) approaches have abilities of autonomous decision making, but time-consuming modeling and training procedures limit their applications. To get over this dilemma, this article proposes the broad reinforcement learning (BRL) approach that fits fast autonomous IoT as it combines the broad learning system (BLS) with a reinforcement learning paradigm to improve the agent's efficiency and accuracy of modeling and decision making. Specifically, a BRL framework is first constructed. Then, the associated learning algorithm, containing training pool introduction, training sample preparation, and incremental learning for BLS, is carefully designed. Finally, as a case study of fast autonomous IoT, the proposed BRL approach is applied to traffic light control, aiming to alleviate traffic congestion in the intersections of smart cities. The experimental results show that the proposed BRL approach can learn better action policy at a shorter execution time when compared with competing approaches.

computer science, information systems,telecommunications,engineering, electrical & electronic
Autonomous Management of Energy-Harvesting IoT Nodes Using Deep Reinforcement Learning

Abdulmajid Murad,Frank Alexander Kraemer,Kerstin Bach,Gavin Taylor

DOI: https://doi.org/10.1109/SASO.2019.00015

2019-05-10

Abstract:Reinforcement learning (RL) is capable of managing wireless, energy-harvesting IoT nodes by solving the problem of autonomous management in non-stationary, resource-constrained settings. We show that the state-of-the-art policy-gradient approaches to RL are appropriate for the IoT domain and that they outperform previous approaches. Due to the ability to model continuous observation and action spaces, as well as improved function approximation capability, the new approaches are able to solve harder problems, permitting reward functions that are better aligned with the actual application goals. We show such a reward function and use policy-gradient approaches to learn capable policies, leading to behavior more appropriate for IoT nodes with less manual design effort, increasing the level of autonomy in IoT.

Machine Learning,Artificial Intelligence,Signal Processing
A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving

Dianzhao Li,Ostap Okhrin

2024-08-28

Abstract:Deep Reinforcement Learning (DRL) has shown remarkable success in solving complex tasks across various research fields. However, transferring DRL agents to the real world is still challenging due to the significant discrepancies between simulation and reality. To address this issue, we propose a robust DRL framework that leverages platform-dependent perception modules to extract task-relevant information and train a lane-following and overtaking agent in simulation. This framework facilitates the seamless transfer of the DRL agent to new simulated environments and the real world with minimal effort. We evaluate the performance of the agent in various driving scenarios in both simulation and the real world, and compare it to human players and the PID baseline in simulation. Our proposed framework significantly reduces the gaps between different platforms and the Sim2Real gap, enabling the trained agent to achieve similar performance in both simulation and the real world, driving the vehicle effectively.

Machine Learning,Artificial Intelligence,Robotics
Autonomous Highway Driving using Deep Reinforcement Learning

Subramanya Nageshrao,Eric Tseng,Dimitar Filev

DOI: https://doi.org/10.48550/arXiv.1904.00035

2019-03-30

Abstract:The operational space of an autonomous vehicle (AV) can be diverse and vary significantly. This may lead to a scenario that was not postulated in the design phase. Due to this, formulating a rule based decision maker for selecting maneuvers may not be ideal. Similarly, it may not be effective to design an a-priori cost function and then solve the optimal control problem in real-time. In order to address these issues and to avoid peculiar behaviors when encountering unforeseen scenario, we propose a reinforcement learning (RL) based method, where the ego car, i.e., an autonomous vehicle, learns to make decisions by directly interacting with simulated traffic. The decision maker for AV is implemented as a deep neural network providing an action choice for a given system state. In a critical application such as driving, an RL agent without explicit notion of safety may not converge or it may need extremely large number of samples before finding a reliable policy. To best address the issue, this paper incorporates reinforcement learning with an additional short horizon safety check (SC). In a critical scenario, the safety check will also provide an alternate safe action to the agent provided if it exists. This leads to two novel contributions. First, it generalizes the states that could lead to undesirable "near-misses" or "collisions ". Second, inclusion of safety check can provide a safe and stable training environment. This significantly enhances learning efficiency without inhibiting meaningful exploration to ensure safe and optimal learned behavior. We demonstrate the performance of the developed algorithm in highway driving scenario where the trained AV encounters varying traffic density in a highway setting.

Robotics,Machine Learning,Systems and Control
Deep Reinforcement Learning framework for Autonomous Driving

Ahmad El Sallab,Mohammed Abdou,Etienne Perot,Senthil Yogamani

DOI: https://doi.org/10.48550/arXiv.1704.02532

IF: 5.414

2017-04-08

Machine Learning

Abstract:Reinforcement learning is considered to be a strong AI paradigm which can be used to teach machines through interaction with the environment and learning from their mistakes. Despite its perceived utility, it has not yet been successfully applied in automotive applications. Motivated by the successful demonstrations of learning of Atari games and Go by Google DeepMind, we propose a framework for autonomous driving using deep reinforcement learning. This is of particular relevance as it is difficult to pose autonomous driving as a supervised learning problem due to strong interactions with the environment including other vehicles, pedestrians and roadworks. As it is a relatively new area of research for autonomous driving, we provide a short overview of deep reinforcement learning and then describe our proposed framework. It incorporates Recurrent Neural Networks for information integration, enabling the car to handle partially observable scenarios. It also integrates the recent work on attention models to focus on relevant information, thereby reducing the computational complexity for deployment on embedded hardware. The framework was tested in an open source 3D car racing simulator called TORCS. Our simulation results demonstrate learning of autonomous maneuvering in a scenario of complex road curvatures and simple interaction of other vehicles.
Deep Attention Driven Reinforcement Learning (DAD-RL) for Autonomous Decision-Making in Dynamic Environment

Jayabrata Chowdhury,Venkataramanan Shivaraman,Sumit Dangi,Suresh Sundaram,P.B.Sujit

2024-09-28

Abstract:Autonomous Vehicle (AV) decision making in urban environments is inherently challenging due to the dynamic interactions with surrounding vehicles. For safe planning, AV must understand the weightage of various spatiotemporal interactions in a scene. Contemporary works use colossal transformer architectures to encode interactions mainly for trajectory prediction, resulting in increased computational complexity. To address this issue without compromising spatiotemporal understanding and performance, we propose the simple Deep Attention Driven Reinforcement Learning (DADRL) framework, which dynamically assigns and incorporates the significance of surrounding vehicles into the ego's RL driven decision making process. We introduce an AV centric spatiotemporal attention encoding (STAE) mechanism for learning the dynamic interactions with different surrounding vehicles. To understand map and route context, we employ a context encoder to extract features from context maps. The spatiotemporal representations combined with contextual encoding provide a comprehensive state representation. The resulting model is trained using the Soft Actor Critic (SAC) algorithm. We evaluate the proposed framework on the SMARTS urban benchmarking scenarios without traffic signals to demonstrate that DADRL outperforms recent state of the art methods. Furthermore, an ablation study underscores the importance of the context-encoder and spatio temporal attention encoder in achieving superior performance.

Artificial Intelligence,Robotics

[Calcification pattern of rat and rabbit incisor dentin].

Toward Enhanced Reinforcement Learning-Based Resource Management via Digital Twin: Opportunities, Applications, and Challenges

Dense reinforcement learning for safety validation of autonomous vehicles

Deep Reinforcement Learning for Online Resource Allocation in IoT Networks: Technology, Development, and Future Challenges

A Digital Twin Framework for Reinforcement Learning with Real-Time Self-Improvement via Human Assistive Teleoperation

Digital twin-based reinforcement learning framework: application to autonomous mobile robot dispatching

Deep Reinforcement Learning (DRL)-based Resource Management in Software-Defined and Virtualized Vehicular Ad Hoc Networks

Resource Constrained Deep Reinforcement Learning

Joint Optimization of Sensing, Decision-making and Motion-controlling for Autonomous Vehicles: A Deep Reinforcement Learning Approach

From Naturalistic Traffic Data to Learning-Based Driving Policy: A Sim-to-Real Study

Deep Reinforcement Learning Enables Joint Trajectory and Communication in Internet of Robotic Things

Data-Driven Distributed Common Operational Picture from Heterogeneous Platforms using Multi-Agent Reinforcement Learning

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Energy Conservation for Internet of Things Tracking Applications Using Deep Reinforcement Learning

Reinforcement Learning from Simulation to Real World Autonomous Driving using Digital Twin

Broad Reinforcement Learning for Supporting Fast Autonomous IoT

Autonomous Management of Energy-Harvesting IoT Nodes Using Deep Reinforcement Learning

A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving

Autonomous Highway Driving using Deep Reinforcement Learning

Deep Reinforcement Learning framework for Autonomous Driving

Deep Attention Driven Reinforcement Learning (DAD-RL) for Autonomous Decision-Making in Dynamic Environment