Abstract:PurposeOne of the promising options for motion management in radiation therapy (RT) is the use of LINAC‐compatible robotic‐arm‐mounted ultrasound imaging system due to its high soft tissue contrast, real‐time capability, absence of ionizing radiation, and low cost. The purpose of this work is to develop a novel deep learning‐based real‐time motion tracking strategy for ultrasound image‐guided RT.MethodsThe proposed tracker combined the attention‐aware fully convolutional neural network (FCNN) and the convolutional long short‐term memory network (CLSTM) that is end‐to‐end trainable. The glimpse sensor module was built inside the attention‐aware FCNN to discard majority of background by focusing on a region containing the object of interest. FCNN extracted discriminating spatial features of glimpse to facilitate temporal modeling for CLSTM. The saliency mask computed from CLSTM refined the features particular to the tracked landmarks. Moreover, the multitask loss strategy including bounding box loss, localization loss, saliency loss, and adaptive loss weighting term was utilized to facilitate training convergence and avoid over/underfitting. The tracker was tested on the databases provided by MICCAI 2015 challenges, and the ground truth data were obtained with the help of brute force‐based template matching technology.ResultsThe mean tracking error of 0.97 ± 0.52 mm and maximum tracking error of 1.94 mm were observed for 85 point landmarks across 39 ultrasound cases compared to the ground truth annotations. The tracking speed per frame per landmark with the GPU implementation ranged from 66 and 101 frames per second, which largely exceeded the ultrasound imaging rate.ConclusionThe results demonstrated the robustness and accuracy of the proposed deep learning‐based motion estimation, despite the existence of some known shortcomings of ultrasound imaging such as speckle noise. The tracking speed of the system was found to be remarkable, sufficiently fast for real‐time applications in RT environment. The approach provides a valuable tool to guide RT treatment with beam gating or multileaf collimator (MLC) tracking in real time.

Unsupervised Convolutional Neural Network for Motion Estimation in Ultrasound Elastography

Convolutional Neural Network-Based Speckle Tracking for Ultrasound Strain Elastography: an Unsupervised Learning Approach.

Global Ultrasound Elastography Using Convolutional Neural Network

An Unsupervised Approach to Ultrasound Elastography with End-to-end Strain Regularisation

Bi-Directional Semi-Supervised Training of Convolutional Neural Networks for Ultrasound Elastography Displacement Estimation

A Fast Normalized Cross-Correlation Calculation Method for Motion Estimation

Deep Motion Network for Freehand 3D Ultrasound Reconstruction

Cascaded one-shot deformable convolutional neural networks: Developing a deep learning model for respiratory motion estimation in ultrasound sequences

A Relative Motion Model for Myocardial Infarction Detection

Deep Learning for Ultrasound Localization Microscopy

Motion Decoupling Network for Intra-Operative Motion Estimation Under Occlusion

Attention-aware Fully Convolutional Neural Network with Convolutional Long Short-Term Memory Network for Ultrasound-Based Motion Tracking

Direct Reconstruction of Ultrasound Elastography Using an End-to-End Deep Neural Network.

Unsupervised Deformable Image Registration for Respiratory Motion Compensation in Ultrasound Images

Deep learning-based motion tracking using ultrasound images

Performance Comparison of Rigid and Affine Models for Motion Estimation Using Ultrasound Radio-Frequency Signals

Automatic Frame Selection Using MLP Neural Network in Ultrasound Elastography

Full Spatial Muscle Fiber Orientation Estimation From Ultrasound Images Using a Multitask Deformable Residual Neural Network

VoxelMorph-Based Deep Learning Motion Correction for Ultrasound Localization Microscopy of Spinal Cord

Robust Fast Inter-Bin Image Registration for Undersampled Coronary MRI Based on a Learned Motion Prior

Neural Multi-Scale Self-Supervised Registration for Echocardiogram Dense Tracking