Abstract:Intra-operative 2D-3D registration of X-ray images with pre-operatively acquired CT scans is a crucial procedure in orthopedic surgeries. Anatomical landmarks pre-annotated in the CT volume can be detected in X-ray images to establish 2D-3D correspondences, which are then utilized for registration. However, registration often fails in certain view angles due to poor landmark visibility. We propose a novel method to address this issue by detecting arbitrary landmark points in X-ray images. Our approach represents 3D points as distinct subspaces, formed by feature vectors (referred to as ray embeddings) corresponding to intersecting rays. Establishing 2D-3D correspondences then becomes a task of finding ray embeddings that are close to a given subspace, essentially performing an intersection test. Unlike conventional methods for landmark estimation, our approach eliminates the need for manually annotating fixed landmarks. We trained our model using the synthetic images generated from CTPelvic1K CLINIC dataset, which contains 103 CT volumes, and evaluated it on the DeepFluoro dataset, comprising real X-ray images. Experimental results demonstrate the superiority of our method over conventional methods. The code is available at <a class="link-external link-https" href="https://github.com/Pragyanstha/rayemb" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

This paper attempts to solve the problem of 2D - 3D registration between X - ray images and pre - operative CT scans, especially in cases where registration fails due to poor visibility of anatomical landmarks from certain perspectives. Specifically, the paper aims to establish 2D - 3D correspondence by detecting arbitrary landmarks in X - ray images, thereby improving the success rate and accuracy of registration. ### Problem Background In orthopedic surgery, intraoperative 2D - 3D registration is a crucial step in aligning X - ray images with pre - operative CT scans. By detecting the annotated anatomical landmarks in the CT volume and finding the corresponding 2D positions in the X - ray image, 2D - 3D correspondence can be established and then used for registration. However, from certain perspectives, traditional registration methods often fail due to poor visibility of the landmarks. ### Method Proposed in the Paper To solve this problem, the paper proposes a new method - RayEmb (Arbitrary Landmark Detection Based on Ray - Embedded Subspace). The main innovations of this method include: 1. **Arbitrary Landmark Detection**: It does not require manual annotation of fixed landmarks but is able to detect arbitrary 3D landmarks in X - ray images. 2. **Ray - Embedded Subspace Representation**: By representing 3D points as a subspace formed by feature vectors (called ray - embeddings), establishing 2D - 3D correspondence becomes the task of finding ray - embeddings close to a given subspace. 3. **Self - Supervised Training Strategy**: It utilizes synthetic data generation for training, avoiding the need for domain experts to annotate landmarks. ### Formula Representation - The formation process of X - ray images is usually described by the Beer - Lambert law: \[ I(d; T)=I_0 \exp\left(-\int_0^x \mu(t_d; T) dt\right) \] where \(I(d)\) is the intensity of X - ray after passing through the material, \(I_0\) is the initial intensity, \(\mu\) is the linear attenuation coefficient, \(d\) is the direction vector pointing to the detector pixel, and \(T\) represents the pose of the volume. - The goal is to minimize the difference between the rendered image and the target image: \[ T_{\text{optim}}=\arg \min_T d(I_{\text{target}}, I(T)) \] - Definition of ray - embedding: \[ e(x)=E(I_q(x); w) \] \[ e'_t(x)=E(I'_t(x); w) \] - Subspace representation: \[ F=(e'_1, e'_2, \ldots, e'_N) \] \[ P = FF^+ = U\Sigma\Sigma^+U^T \] - Estimation of corresponding 2D landmarks: \[ \hat{x}=\arg \max_x \text{sim}(P, x) \] where \[ \text{sim}(P, x)=\frac{e^T(x) P e(x)}{|e^T(x)||P e(x)|} \] - InfoNCE loss function: \[ L(e, e'_t)=-\log \frac{\exp(\text{sim}(P, x_+)/\tau)}{\sum_{x_- \in N} \exp(\text{sim}(P, x_-)/\tau)} \] ### Summary The RayEmb method proposed in the paper solves the problem of poor visibility of landmarks in certain perspectives in traditional 2D - 3D registration methods by introducing ray - embedded subspace representation and self - supervised training strategy, improving the success rate and accuracy of registration. Experimental results show that this method outperforms traditional methods in multiple test scenarios.

RayEmb: Arbitrary Landmark Detection in X-Ray Images Using Ray Embedding Subspace

Automatic X-ray Landmark Detection and Shape Segmentation Via Data-Driven Joint Estimation of Image Displacements.

Segmentation-Guided Coordinate Regression for Robust Landmark Detection on X-Rays: Application to Automated Assessment of Lower Limb Alignment

CephalFormer: Incorporating Global Structure Constraint into Visual Features for General Cephalometric Landmark Detection

Fully-Automatic Landmark detection in Skull X-Ray images

LandmarkNet: a 2D Digital Radiograph Landmark Estimator for Registration

PELE scores: pelvic X-ray landmark detection with pelvis extraction and enhancement

Multi-View Fusion Convolutional Neural Network for Automatic Landmark Location on Spinal X-Rays.

Spatial-Division Augmented Occupancy Field for Bone Shape Reconstruction from Biplanar X-Rays

Multi-Scale Semantic Fusion of a Large Receptive Field for Irregular Pelvic X-Ray Landmark Detection

Personalized X-ray Reconstruction of the Proximal Femur Via a New Control Point-based 2D-3D Registration and Residual Complexity Minimization.

A machine learning pipeline for internal anatomical landmark embedding based on a patient surface model

Transfer learning from an artificial radiograph-landmark dataset for registration of the anatomic skull model to dual fluoroscopic X-ray images

Fully Automatic X-Ray Image Segmentation Via Joint Estimation Of Image Displacements

Non-Rigid 2D-3D Registration Using Convolutional Autoencoders

Structure-Aware Sparse-View X-ray 3D Reconstruction

Multi-target landmark detection with incomplete images via reinforcement learning and shape prior embedding

An Attention-Guided Deep Regression Model for Landmark Detection in Cephalograms

Part Affinity Fields and CoordConv for Detecting Landmarks of Lumbar Vertebrae and Sacrum in X-ray Images

Benchmarking Encoder-Decoder Architectures for Biplanar X-ray to 3D Shape Reconstruction

Fully Automatic Cephalometric X-Ray Landmark Detection Using Random Forest Regression and Sparse shape composition