Abstract:We propose Deep-Motion-Net: an end-to-end graph neural network (GNN) architecture that enables 3D (volumetric) organ shape reconstruction from a single in-treatment kV planar X-ray image acquired at any arbitrary projection angle. Estimating and compensating for true anatomical motion during radiotherapy is essential for improving the delivery of planned radiation dose to target volumes while sparing organs-at-risk, and thereby improving the therapeutic ratio. Achieving this using only limited imaging available during irradiation and without the use of surrogate signals or invasive fiducial markers is attractive. The proposed model learns the mesh regression from a patient-specific template and deep features extracted from kV images at arbitrary projection angles. A 2D-CNN encoder extracts image features, and four feature pooling networks fuse these features to the 3D template organ mesh. A ResNet-based graph attention network then deforms the feature-encoded mesh. The model is trained using synthetically generated organ motion instances and corresponding kV images. The latter is generated by deforming a reference CT volume aligned with the template mesh, creating digitally reconstructed radiographs (DRRs) at required projection angles, and DRR-to-kV style transferring with a conditional CycleGAN model. The overall framework was tested quantitatively on synthetic respiratory motion scenarios and qualitatively on in-treatment images acquired over full scan series for liver cancer patients. Overall mean prediction errors for synthetic motion test datasets were 0.16$\pm$0.13 mm, 0.18$\pm$0.19 mm, 0.22$\pm$0.34 mm, and 0.12$\pm$0.11 mm. Mean peak prediction errors were 1.39 mm, 1.99 mm, 3.29 mm, and 1.16 mm.

Image-to-Graph Convolutional Network for Deformable Shape Reconstruction from a Single Projection Image

IGCN: Image-to-graph Convolutional Network for 2D/3D Deformable Registration

Intrinsic Image Transformation via Scale Space Decomposition

Deep Fusion Network Based Sparse View CT Reconstructions for Clinical Diagnostic Scanners

Deep-Motion-Net: GNN-based volumetric organ shape reconstruction from single-view 2D projections

Statistical Deformation Reconstruction Using Multi-organ Shape Features for Pancreatic Cancer Localization

Image-Guided Human Reconstruction via Multi-Scale Graph Transformation Networks

End-to-end Deformable Attention Graph Neural Network for Single-view Liver Mesh Reconstruction

DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image

Shape Reconstruction With Multiphase Conductivity for Electrical Impedance Tomography Using Improved Convolutional Neural Network Method

DSC-Recon: Dual-Stage Complementary 4D Organ Reconstruction from X-ray Image Sequence for Intraoperative Fusion

Embedding-Alignment Fusion-Based Graph Convolution Network With Mixed Learning Strategy for 4D Medical Image Reconstruction

Task-Specific Data Preparation for Deep Learning to Reconstruct Structures of Interest from Severely Truncated CBCT Data

Deep Learning-Based Reconstruction of Interventional Tools from Four X-Ray Projections for Tomographic Interventional Guidance

Shape registration with learned deformations for 3D shape reconstruction from sparse and incomplete point clouds

X2V: 3D Organ Volume Reconstruction From a Planar X-Ray Image With Neural Implicit Methods

Attention-based Shape-Deformation Networks for Artifact-Free Geometry Reconstruction of Lumbar Spine from MR Images

ReshapeIT: Reliable Shape Interaction with Implicit Template for Anatomical Structure Reconstruction

End-To-End Convolutional Neural Network for 3D Reconstruction of Knee Bones From Bi-Planar X-Ray Images

High-Quality Textured 3D Shape Reconstruction with Cascaded Fully Convolutional Networks

Enhancement of 4-D Cone-Beam Computed Tomography (4D-CBCT) Using a Dual-Encoder Convolutional Neural Network (Decnn)