Abstract:Purpose: The accurate detection and tracking of devices, such as guiding catheters in live X-ray image acquisitions, are essential prerequisites for endovascular cardiac interventions. This information is leveraged for procedural guidance, e.g., directing stent placements. To ensure procedural safety and efficacy, there is a need for high robustness/no failures during tracking. To achieve this, one needs to efficiently tackle challenges, such as device obscuration by the contrast agent or other external devices or wires and changes in the field-of-view or acquisition angle, as well as the continuous movement due to cardiac and respiratory motion. Approach: To overcome the aforementioned challenges, we propose an approach to learn spatio-temporal features from a very large data cohort of over 16 million interventional X-ray frames using self-supervision for image sequence data. Our approach is based on a masked image modeling technique that leverages frame interpolation-based reconstruction to learn fine inter-frame temporal correspondences. The features encoded in the resulting model are fine-tuned downstream in a light-weight model. Results: Our approach achieves state-of-the-art performance, in particular for robustness, compared to ultra optimized reference solutions (that use multi-stage feature fusion or multi-task and flow regularization). The experiments show that our method achieves a 66.31% reduction in the maximum tracking error against the reference solutions (23.20% when flow regularization is used), achieving a success score of 97.95% at a 3× faster inference speed of 42 frames-per-second (on GPU). In addition, we achieve a 20% reduction in the standard deviation of errors, which indicates a much more stable tracking performance. Conclusions: The proposed data-driven approach achieves superior performance, particularly in robustness and speed compared with the frequently used multi-modular approaches for device tracking. The results encourage the use of our approach in various other tasks within interventional image analytics that require effective understanding of spatio-temporal semantics.

Detection of Thermal Events by Semi-Supervised Learning for Tokamak First Wall Safety

Detection of Thermal Events by Semi-Supervised Learning for Tokamak First Wall Safety

Using convolutional neural networks to detect edge localized modes in DIII-D from Doppler backscattering measurements

Tokamak divertor plasma emulation with machine learning

A system diagnostic and prognostic framework based on deep learning for advanced reactors

Application of machine learning for detecting and tracking turbulent structures in plasma fusion devices using ultra fast imaging

A machine-learning-based tool for last closed-flux surface reconstruction on tokamaks

Identifying L-H transition in HL-2A through deep learning

Using a Physics Constrained U-Net for Real-Time Compatible Extraction of Physical Features from WEST Divertor Hot-Spots

eXplainable artificial intelligence applied to algorithms for disruption prediction in tokamak devices

A machine-learning-aided data recovery approach for predicting multi-material thermal behaviors in advanced test reactor capsules

A hybrid physics/data-driven logic to detect, classify, and predict anomalies and disruptions in tokamak plasmas

Application of interpretable machine learning for cross-diagnostic inference on the ST40 spherical tokamak

Classification in Early Fire Detection Using Multi-Sensor Nodes—A Transfer Learning Approach

Classification of the L-, H-mode, and plasma-free state: Convolutional neural networks and variational autoencoders on the edge reflectometer for KSTAR

Adaptive Anomaly Detection Disruption Prediction Starting from First Discharge on Tokamak

A Semisupervised Learning Framework for Recognition and Classification of Defects in Transient Thermography Detection

Self-supervised learning for interventional image analytics: toward robust device trackers

Hybrid Twins Modeling of a High-Level Radioactive Waste Cell Demonstrator for Long-Term Temperature Monitoring and Forecasting

Avoiding fusion plasma tearing instability with deep reinforcement learning