[Case of thorotrast-induced canceer of the liver with epidural metastasis to the spinal cord].

T. Yanagi,S. Suga,H. Mekata,T. Yoshida,H. Maeda

1969-10-01

Abstract:

What problem does this paper attempt to address?

Dynamic Submodular-Based Learning Strategy in Imbalanced Drifting Streams for Real-Time Safety Assessment in Nonstationary Environments.

Zeyi Liu,Xiao He

DOI: https://doi.org/10.1109/tnnls.2023.3294788

IF: 14.255

2023-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:The design of real-time safety assessment (RTSA) approaches in nonstationary environments is meaningful to reduce the possibility of significant losses. However, several challenging problems are needed to be well considered. The performance of existing approaches will be negatively affected in the settings of imbalanced drifting streams. In this case, the model design with the incremental update should also be explored. Furthermore, the query strategy should also be well-designed. This article investigates a dynamic submodular-based learning strategy to address such issues. Specifically, an efficient incremental update procedure is designed with the structure of the broad learning system (BLS), which is beneficial to the detection of concept drift. Furthermore, a novel dynamic submodular-based annotation with an activation interval strategy is proposed to select valuable samples in imbalanced drifting streams. The lower bound of annotation value is also proven theoretically with a novel drift adaption mechanism. Numerous experiments are conducted with the realistic data of JiaoLong deep-sea manned submersible. The experimental results show that the proposed approach can achieve better assessment accuracy than typical existing approaches.
Online Dynamic Hybrid Broad Learning System for Real-Time Safety Assessment of Dynamic Systems

Zeyi Liu,Xiao He

DOI: https://doi.org/10.1109/tkde.2024.3475028

IF: 9.235

2024-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Real-time safety assessment of dynamic systems is of paramount importance in industrial processes since it provides continuous monitoring and evaluation to prevent potential harm to the environment and individuals. However, there are still several challenges to be resolved due to the requirements of time consumption and the non-stationary nature of real-world environments. In this paper, a novel online dynamic hybrid broad learning system, termed ODH-BLS, is proposed to more fully utilize the co-design advantages of active adaptation and passive adaptation. It makes effective use of limited annotations with the proposed sample value function. Simultaneously, anchor points can be dynamically adjusted to accommodate changes of the underlying distribution, thereby leveraging the value of unlabeled samples. An iterative update rule is also derived to ensure adaptation of the assessment model to real-time data at low computational costs. We also provide theoretical analyses to illustrate its practicality. Several experiments regarding the JiaoLong deep-sea manned submersible are carried out. The results demonstrate that the proposed ODH-BLS method achieves a performance improvement of approximately 8% over the baseline method on the benchmark dataset, showing its effectiveness in solving real-time safety assessment tasks for dynamic systems.
An Online Active Broad Learning Approach for Real-Time Safety Assessment of Dynamic Systems in Nonstationary Environments

Zeyi Liu,Yi Zhang,Zhongjun Ding,Xiao He

DOI: https://doi.org/10.1109/tnnls.2022.3222265

IF: 14.255

2022-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Real-time safety assessment of the complex dynamic systems in nonstationary environments is of great significance for avoiding the potential hazards. In this case, the update procedure with high assessment accuracy and training speed is crucial and meaningful in the dynamic streaming setting. Generally, the performance of most online learning approaches will be negatively affected by limited annotated samples in such a setting. Moreover, the time cost of advanced conventional methods with retaining procedures is relatively high, constraining their practicality. In this article, a novel online active broad learning approach, termed OABL, is proposed. In detail, the effectiveness of the broad learning system in the framework of online active learning is first revealed and verified. A reasonable dynamic asymmetric query strategy is then designed with a limited annotation budget to actively annotate the relatively valuable samples, which is beneficial to mitigating the negative effects of class imbalance. In this context, the advantage of the human-in-the-loop characteristic is also effectively used to control the evolution direction of the learner during the incremental update, which makes it better able to adapt to complex and nonstationary environments. Several related experiments are conducted with the realistic data of JiaoLong deep-sea manned submersible. Results show the effectiveness and practicality of the proposal compared with the existing advanced approaches.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
A Deep Learning Method for Lane Changing Situation Assessment and Decision Making.

Xiao Liu,Jun Liang,Bing Xu

DOI: https://doi.org/10.1109/access.2019.2940853

IF: 3.9

2019-01-01

IEEE Access

Abstract:Compared with the lane keeping maneuvers, lane changing maneuvers are much more complicated. Inappropriate ones may result in traffic accidents. Therefore, it is necessary to develop vehicular social networks(VSNs) and advanced driver assistance systems(ADAS), which can help to evaluate the traffic situation and provide additional warnings for drivers' unsafe lane changing operations. This paper proposes a deep learning model to simulate the situation assessment and decision making process during lane changing events. Compared with the existing assessment models, our model has two significant advantages. First, except for the instantaneous states of the subject and the surrounding vehicles, the proposed model also takes drivers' historical experience and the vehicle-to-vehicle (V2V) memory effect into consideration for the final lane changing maneuvers situation assessment. Second, our new assessment model is built based on Deep Neural Networks(DNNs), which is proved to outperform conventional machine learning classifiers. The empirical trajectory dataset NGSIM is used for evaluating the performance of the proposed model. Experiment results indicate that our model achieves high identification accuracy of both lane changing maneuvers and lane keeping maneuvers. It is verified to be an effective driverless technology in assisting vehicle warning systems.
Real-Time Safety Assessment for Dynamic Systems with Limited Memory and Annotations

Zeyi Liu,Xiao He

DOI: https://doi.org/10.1109/tits.2023.3266256

IF: 8.5

2023-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:Real-time safety assessment of dynamic systems has recently received increasing attention. However, the performance of existing advanced approaches is often negatively affected by realistic requirements such as limited annotations and memory. In this case, how to design reasonable query strategies to select valuable instances and exploit the memory space efficiently is extremely meaningful. This paper proposes a novel memory-triggered submodularity-guided active broad learning approach, termed MTSGABL, to deal with such issues simultaneously. Specifically, the broad learning system is introduced as the basic assessment model to update incrementally. A memory-triggered learning mechanism is then proposed based on the drift detection procedure, which controls the update process to exploit the latest sequential information. Furthermore, a submodularity-guided query strategy is introduced to select a small number of valuable samples sequentially, which is beneficial to alleviate the negative effects of the imbalanced data stream. Numerous comparison and ablation experiments with the realistic JiaoLong deep-sea manned submersible data are conducted to validate its effectiveness. Results show that the proposed approach is superior to the existing typical approaches subject to these constraints.
Expert-augmented Data-driven Safety Level Assessment Scheme with Incremental Learning.

Chang Liu,Yi Zhang,Xiao He

DOI: https://doi.org/10.1109/SAFEPROCESS52771.2021.9693634

2021-01-01

Abstract:In this paper, the data-driven safety level assessment problem for dynamic systems with only normal scenario data is studied, i.e. no abnormal scenarios data are available. An expert-augmented data-driven safety level assessment framework is proposed, which combines expert knowledge and data-driven algorithms to tackle the problem. The user-defined high-level description of the system is introduced firstly, which consists of the attribute that denotes specific subsystem is abnormal or not. In this way, expert knowledge can be represented as the attributed description prototype matrix with corresponding safety levels. Considering the data is non-stationary and usually comes in the stream, the fast incremental support vector data description (FISVDD) algorithm, which can be trained incrementally, is used to detect the attribute description based on the online measured variables. At last, the nearest neighbor search is used to determine the safety level by calculating and sorting similarities between the attribute description and prototypes. A practical case study based on JiaoLong deep-sea manned submersible demonstrates the effectiveness of the proposed scheme.
Contrastive Preference-Guided Active Learning Approach Based on Ranking Correlation for Real-Time Safety Assessment

Zeyi Liu,Xiao He

DOI: https://doi.org/10.1109/tase.2024.3401470

IF: 6.636

2024-01-01

IEEE Transactions on Automation Science and Engineering

Abstract:Real-time safety assessment is a critical process for identifying and analyzing potential safety hazards in industrial applications. Active learning has been widely recognized as an effective technique for addressing such issues by utilizing small labeled samples to achieve high evaluation performance. However, in real-world scenarios, the performance of query strategies can be significantly impacted by the data environment. Therefore, it is crucial to co-design a strategy using different criteria to ensure reliable and stable safety assessment results for system operation. In this paper, we propose a contrastive preference-guided active learning approach to tackle chunk-level real-time safety assessment tasks in non-stationary environments. Firstly, we construct negative ranking lists and random lists. Then, we introduce Jeffrey divergence to measure pairwise ranking correlation. By leveraging contrastive preference relationships, we can effectively obtain the value of samples in data chunks with preference aggregation procedures. To verify the effectiveness of the proposed method, we conduct numerous experiments using realistic data from the JiaoLong deep-sea manned submersible. The results demonstrate that our approach outperforms most existing advanced methods in terms of performance stability and accuracy. Note to Practitioners —In situations where a significant number of samples must be processed concurrently to promptly identify and mitigate potential hazards, it is vital to consider the responsibilities of real-time safety assessment (RTSA). The proposed approach offers a resolution to the difficulty of acquiring annotations for all samples due to the continual stream of data. Multiple query criteria can be seamlessly integrated, which enhances the stability and superiority of learning performance simultaneously. The approach’s scalability, flexibility, and effectiveness render it a valuable instrument for guaranteeing a prompt response to emergent procedural threats and safety. It can accommodate any number and type of advanced single query criteria, making it easy to adapt to various scenarios. The proposed CPRC approach signifies a significant breakthrough in the field of real-time safety assessment, with the potential to enhance safety outcomes in industrial settings. Multiple experiments involving the realistic JiaoLong deep-sea manned submersible were conducted, and the outcomes demonstrate the benefits of this approach for practical RTSA applications.
Improving the Generalization Performance of Data-driven Predictive Model for Dynamic Process Systems

Weiming Shao,Yougao Li,Dongya Zhao,Zhiqiang Ge

DOI: https://doi.org/10.1109/ddcls52934.2021.9455521

2021-01-01

Abstract:Data-driven dynamic predictive models have been extensively applied in dynamic process systems for online predicting difficult-to-measure key variables. However, due to the limitations of industrial data in quantity, quality and representativity, the generalization performance of the developed dynamic models are not reliable. In other words, based on certain unique historical dataset, high-accuracy of the predictive model for online unseen data cannot be guaranteed. To deal with this issue, this paper proposes a switching model-based dynamic modeling framework for improving the generalization accuracy and reliability of data-driven predictive models, which is composed of two stages, i.e., the offline model library construction and online model switch. The performance of the proposed scheme is assessed by a real-life industrial dynamic process using switching dynamic partial least squares models, and its application potentials are verified through comparisons among the state-of-the-art methods.
Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention

Yunkun Xu,Zhenyu Liu,Guifang Duan,Jiangcheng Zhu,Xiaolong Bai,Jianrong Tan

DOI: https://doi.org/10.48550/arxiv.2111.05819

2021-01-01

Abstract:Safety has become one of the main challenges of applying deep reinforcement learning to real world systems. Currently, the incorporation of external knowledge such as human oversight is the only means to prevent the agent from visiting the catastrophic state. In this paper, we propose MBHI, a novel framework for safe model-based reinforcement learning, which ensures safety in the state-level and can effectively avoid both"local"and"non-local"catastrophes. An ensemble of supervised learners are trained in MBHI to imitate human blocking decisions. Similar to human decision-making process, MBHI will roll out an imagined trajectory in the dynamics model before executing actions to the environment, and estimate its safety. When the imagination encounters a catastrophe, MBHI will block the current action and use an efficient MPC method to output a safety policy. We evaluate our method on several safety tasks, and the results show that MBHI achieved better performance in terms of sample efficiency and number of catastrophes compared to the baselines.
CADM$+$: Confusion-Based Learning Framework With Drift Detection and Adaptation for Real-Time Safety Assessment

Songqiao Hu,Zeyi Liu,Minyue Li,Xiao He

DOI: https://doi.org/10.1109/tnnls.2024.3369315

IF: 14.255

2024-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Real-time safety assessment (RTSA) of dynamic systems holds substantial implications across diverse fields, including industrial and electronic applications. However, the complexity and rapid flow nature of data streams, coupled with the expensive label cost and pose significant challenges. To address these issues, a novel confusion-based learning framework, termed confusion-and-detection method plus (CADM + ), is proposed in this article. When drift occurs, the model is updated with uncertain samples, which may cause confusion between existing and new concepts, resulting in performance differences. The cosine similarity is used to measure the degree of such conceptual confusion in the model. Furthermore, the change of standard deviation within a fixed-size cosine similarity window is introduced as an indicator for drift detection. Theoretical demonstrations show the asymptotic increase of cosine similarity. In addition, the approximate independence of the change in standard deviation with the number of trained samples is indicated. Finally, the extreme value theory (EVT) is applied to determine the threshold of judging drifts. Several experiments are conducted to verify its effectiveness. Experimental results prove that the proposed framework is more suitable for RTSA tasks compared with state-of-the-art algorithms. The source code is available at https://github.com/THUFDD/CADM-plus.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Enhancing the Reliability and Accuracy of Data-Driven Dynamic Soft Sensor Based on Selective Dynamic Partial Least Squares Models

Weiming Shao,Wenxue Han,Yougao Li,Zhiqiang Ge,Dongya Zhao

DOI: https://doi.org/10.1016/j.conengprac.2022.105292

IF: 4.057

2022-01-01

Control Engineering Practice

Abstract:Data-driven soft sensors have been widely applied to a broad range of process industries for virtually sensing difficult-to-measure but of-great-concern variables. However, it is still nontrivial to develop dynamic soft sensors with satisfactory performance. A crucial obstacle lies in strong dependence on quality and representativity of collected data. In other words, the predictive accuracy of the developed soft sensor could be quite sensitive to both offline training data and online unseen data, leading to unreliable and degraded generalization performance. In order to deal with such a troublesome issue, this paper proposes a framework for developing reliable dynamic soft sensor called selective dynamic partial least squares (SDPLS). The SDPLS consists of two-stage operations. At the offline stage, aided by intelligent optimization algorithm a model library is established through constructing various DPLS models, each of which accounts for certain ‘mode’. At the online stage, adaptive online model updating for adapting to the current working condition is carried out based on evaluating the performance of the stored individual models, where a correction scheme is also developed for bias elimination. Extensive case studies have been conducted based on a numerical example and two real-life industrial processes, and the results efficaciously demonstrate the effectiveness and promising application prospects of the proposed schemes.
Learning-Based Neural Dynamic Surface Predictive Control for MMC

Xing Liu,Lin Qiu,Jose Rodriguez,Kui Wang,Yongdong Li,Youtong Fang

DOI: https://doi.org/10.1109/tpel.2022.3200857

IF: 5.967

2023-01-01

IEEE Transactions on Power Electronics

Abstract:Reinforcement learning technique was developed recently as an interesting topic in designing adaptive optimal controllers. This technique explicitly provided a feasible solution to circumvent the “curse of dimensionality” and requiring a system model inherent in the classical dynamic programming algorithm. By virtue of this property, in our work, by introducing this technique into a predictor-based online adaptive neural dynamic surface predictive control architecture, we concentrate on a novel robust predictive control framework subject to system uncertainties. To be specific, in this presented framework, an adaptive dynamic programming control strategy utilizing a critic neural network point of view is developed to learn the optimal control policy. Our modification is able to facilitate the alleviation of performance deterioration caused by system uncertainties and enable the smooth and fast learning, while keeping the merits of the finite control-set model predictive control. Finally, the interest and applicability of the proposed control methodology are verified by performance evaluation.
Active Labeling Aided Semi-Supervised Safety Assessment with Task-Related Unknown Scenarios

Chang Liu,Xiao He,Minyue Li,Yi Zhang,Zhongjun Ding

DOI: https://doi.org/10.1109/tr.2024.3376601

IF: 5.883

2024-01-01

IEEE Transactions on Reliability

Abstract:The open environment presents a challenging issue for the online safety assessment of dynamic systems, which means that unknown scenarios may arise unexpectedly. These unknown scenarios can be task-related and result in the within-class distribution mismatch. Addressing this challenge in the semi-supervised safety assessment task, where unlabeled training data contain task-related unknown scenarios, has not been explored. This article investigates this new semi-supervised safety assessment problem. A novel active-labeling-aided semi-supervised learning scheme is proposed to tackle the within-class distribution mismatch between labeled and unlabeled training data. The proposed scheme begins by detecting out-of-distribution unlabeled data through the construction of a deep support vector data description network for each class. Subsequently, an active labeling approach along with its kernel extension is introduced, taking into account both distribution mismatch degree and sample representativeness. The proposed active labeling approach can be seamlessly integrated with any semi-supervised learning algorithm to enhance its performance in handling task-related unknown scenarios. The effectiveness and applicability of the proposed method are demonstrated through case studies based on a bearing dataset and operation data from an actual deep-sea manned submersible.
Active Incremental Learning for Health State Assessment of Dynamic Systems with Unknown Scenarios

Chang Liu,Yi Zhang,Zhongjun Ding,Xiao He

DOI: https://doi.org/10.1109/tii.2022.3181187

IF: 12.3

2023-01-01

IEEE Transactions on Industrial Informatics

Abstract:This article is concerned with the data-driven health state assessment task with unknown scenarios. Unknown scenarios are inaccessible in the stage of model training but they can appear unexpectedly during the running stage. A new problem called the within-class distribution mismatch is raised by assuming that unknown scenarios still belong to known classes. To tackle this challenging problem, a novel active incremental learning scheme with a classifier and an out-of-distribution (OOD) detector is proposed. A k -fast incremental support vector data description ( k -FISVDD) model is put forward as the OOD detector to recognize distribution mismatch samples online for label annotation. Specifically, it integrates the clustering algorithm to build local support vector sets, based on which an active query strategy is developed. An incremental learning mechanism is also designed to reduce the labeling cost. Then, the new labeled data can simultaneously refine the classifier and OOD detector. Two cases, including a bearing benchmark dataset and the operation data of a practical deep-sea manned submersible, are studied to demonstrate the effectiveness of the proposed scheme.
Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance

Wei Guo,Marc Brittain,Peng Wei

DOI: https://doi.org/10.48550/arXiv.2105.02331

2022-02-20

Abstract:The separation assurance task will be extremely challenging for air traffic controllers in a complex and high density airspace environment. Deep reinforcement learning (DRL) was used to develop an autonomous separation assurance framework in our previous work where the learned model advised speed maneuvers. In order to improve the safety of this model in unseen environments with uncertainties, in this work we propose a safety module for DRL in autonomous separation assurance applications. The proposed module directly addresses both model uncertainty and state uncertainty to improve safety. Our safety module consists of two sub-modules: (1) the state safety sub-module is based on the execution-time data augmentation method to introduce state disturbances in the model input state; (2) the model safety sub-module is a Monte-Carlo dropout extension that learns the posterior distribution of the DRL model policy. We demonstrate the effectiveness of the two sub-modules in an open-source air traffic simulator with challenging environment settings. Through extensive numerical experiments, our results show that the proposed sub-safety modules help the DRL agent significantly improve its safety performance in an autonomous separation assurance task.

Artificial Intelligence
How Safe Am I Given What I See? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy

Zhenjiang Mao,Carson Sobolewski,Ivan Ruchkin

2024-06-19

Abstract:End-to-end learning has emerged as a major paradigm for developing autonomous systems. Unfortunately, with its performance and convenience comes an even greater challenge of safety assurance. A key factor of this challenge is the absence of the notion of a low-dimensional and interpretable dynamical state, around which traditional assurance methods revolve. Focusing on the online safety prediction problem, this paper proposes a configurable family of learning pipelines based on generative world models, which do not require low-dimensional states. To implement these pipelines, we overcome the challenges of learning safety-informed latent representations and missing safety labels under prediction-induced distribution shift. These pipelines come with statistical calibration guarantees on their safety chance predictions based on conformal prediction. We perform an extensive evaluation of the proposed learning pipelines on two case studies of image-controlled systems: a racing car and a cartpole.

Machine Learning
Human risk recognition and prediction in manned submersible diving tasks driven by deep learning models

Yidan Qiao,Haotian Li,Dengkai Chen,Hang Zhao,Lin Ma,Yao Wang

DOI: https://doi.org/10.1016/j.aei.2024.102893

IF: 8.8

2024-10-27

Advanced Engineering Informatics

Abstract:The complexity of human cognition is increased by the multiple and interactive information clusters brought about by the application of advanced intelligent information technologies. Especially in systems operating in extreme regions far from society, human errors are more pronounced than ever before. Prolonged social isolation, extreme weightless or overweight environments, stressful atmospheres, and lack of situational awareness are all added potential elements contributing to human risk. Although the development of human reliability analysis methods and their variants continues to mature, accurately predicting the potential risk of dynamic human behavior from sparse and discrete events remains a great challenge. We focus on deep learning computational architectures that are similar to the cognitive processes and mechanisms of the brain, and build neural networks that match the perceptual activation and memory cycling of the cognitive features of the brain. This study focuses on investigating the ability of the joint SNN-ITLSTM network to predict human error behavior and the clusters of performance shaping factors that effectively characterize the far-social nature. Combining the bionic properties of SNN and the temporal update mechanism of LSTM in the form of hierarchical events constitutes a computationally efficient network architecture. Our results show that the joint model proposed in this study has the performance to strengthen temporal influences and characterize cognitive features of the brain.

engineering, multidisciplinary,computer science, artificial intelligence
Discriminator-Guided Model-Based Offline Imitation Learning

Wenjia Zhang,Haoran Xu,Haoyi Niu,Peng Cheng,Ming Li,Heming Zhang,Guyue Zhou,Xianyuan Zhan

DOI: https://doi.org/10.48550/arxiv.2207.00244

2022-01-01

Abstract:Offline imitation learning (IL) is a powerful method to solve decision-making problems from expert demonstrations without reward labels. Existing offline IL methods suffer from severe performance degeneration under limited expert data. Including a learned dynamics model can potentially improve the state-action space coverage of expert data, however, it also faces challenging issues like model approximation/generalization errors and suboptimality of rollout data. In this paper, we propose the Discriminator-guided Model-based offline Imitation Learning (DMIL) framework, which introduces a discriminator to simultaneously distinguish the dynamics correctness and suboptimality of model rollout data against real expert demonstrations. DMIL adopts a novel cooperative-yet-adversarial learning strategy, which uses the discriminator to guide and couple the learning process of the policy and dynamics model, resulting in improved model performance and robustness. Our framework can also be extended to the case when demonstrations contain a large proportion of suboptimal data. Experimental results show that DMIL and its extension achieve superior performance and robustness compared to state-of-the-art offline IL methods under small datasets.
Fast Sensitivity-Analysis-Based Online Self-Organizing Broad Learning System

Ling Yi,Jinliang Ding,Changxin Liu,Tianyou Chai

DOI: https://doi.org/10.1109/tii.2024.3361017

IF: 12.3

2024-01-01

IEEE Transactions on Industrial Informatics

Abstract:Modern industrial process modeling requires models to adapt quickly to real-time operating conditions. To this end, this article proposes a fast sensitivity analysis (SA)-based self-organizing broad learning system (SASO-BLS) that offers a paradigm for theory-guided online structural self-adaptation of differentiable models. Specifically, SASO-BLS is implemented by embedding in BLS an interpretable and efficient model compression method called fast partial differential-based SA (FPD-SA). Unlike conventional SA methods that require iterative evaluation of SA indexes for all samples, FPD-SA exploits the deduced chain rule across categories, effectively mitigating the computational burden imposed by industrial Big Data and reducing computational time. In addition, we derive the offline SASO-BLS algorithm for discrete data and extend it to the online scenario for real-time streaming data processing. Note that both modes obviate the necessity of recalculating the pseudo-inverse of the entire state matrix, facilitating SASO-BLS in attaining remarkable efficiency in structural self-organization. Finally, a theoretical justification of the universal approximation property for SASO-BLS is presented. Experimental results on a fault diagnosis benchmark dataset and a real industrial process one demonstrate the effectiveness of the proposed approach.

automation & control systems,computer science, interdisciplinary applications,engineering, industrial
Dynamics Adaptive Safe Reinforcement Learning with a Misspecified Simulator

Ruiqi Xue,Ziqian Zhang,Lihe Li,Feng Chen,Yi-Chen Li,Yang Yu,Lei Yuan

DOI: https://doi.org/10.1007/978-3-031-70368-3_5

2024-01-01

Abstract:Sim-to-real reinforcement learning offers the advantage of learning safe policies within simulators, circumventing the need for costly trial-and-error in the real world. Traditional approaches often rest on the assumption of consistent state-action transition between the simulator and the real-world environment. However, this assumption can be violated due to the poor fidelity of simulators, leading to a constrained trust region for effective policy learning. The limitation can be more pronounced when safety issues are considered, potentially resulting in threatening policies if no safe samples exist in the trust region. To overcome these challenges, we propose Dynamics Adaptive Safe Reinforcement Learning with a Misspecified Simulator (DASaR). Our approach begins by relaxing the assumption to expand the trust region and theoretically demonstrate the unbounded performance gap inherent in traditional methods. Subsequently, DASaR aligns the estimated value functions in the simulator and the real-world environment via inverse dynamics-based relabeling of reward and cost signals. Furthermore, to deal with the underestimation of cost value functions, DASaR employs uncertainty estimation to improve its conservatism, ensuring the safety of the learned policy. Experiments in various complex environments thoroughly demonstrate DASaR's outstanding ability to balance safety satisfaction and reward maximization across diverse dynamics gaps.

[Case of thorotrast-induced canceer of the liver with epidural metastasis to the spinal cord].

Dynamic Submodular-Based Learning Strategy in Imbalanced Drifting Streams for Real-Time Safety Assessment in Nonstationary Environments.

Online Dynamic Hybrid Broad Learning System for Real-Time Safety Assessment of Dynamic Systems

An Online Active Broad Learning Approach for Real-Time Safety Assessment of Dynamic Systems in Nonstationary Environments

A Deep Learning Method for Lane Changing Situation Assessment and Decision Making.

Real-Time Safety Assessment for Dynamic Systems with Limited Memory and Annotations

Expert-augmented Data-driven Safety Level Assessment Scheme with Incremental Learning.

Contrastive Preference-Guided Active Learning Approach Based on Ranking Correlation for Real-Time Safety Assessment

Improving the Generalization Performance of Data-driven Predictive Model for Dynamic Process Systems

Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention

CADM$+$: Confusion-Based Learning Framework With Drift Detection and Adaptation for Real-Time Safety Assessment

Enhancing the Reliability and Accuracy of Data-Driven Dynamic Soft Sensor Based on Selective Dynamic Partial Least Squares Models

Learning-Based Neural Dynamic Surface Predictive Control for MMC

Active Labeling Aided Semi-Supervised Safety Assessment with Task-Related Unknown Scenarios

Active Incremental Learning for Health State Assessment of Dynamic Systems with Unknown Scenarios

Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance

How Safe Am I Given What I See? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy

Human risk recognition and prediction in manned submersible diving tasks driven by deep learning models

Discriminator-Guided Model-Based Offline Imitation Learning

Fast Sensitivity-Analysis-Based Online Self-Organizing Broad Learning System

Dynamics Adaptive Safe Reinforcement Learning with a Misspecified Simulator