Domain adaptation strategies for 3D reconstruction of the lumbar spine using real fluoroscopy data

Sascha Jecklin,Youyang Shen,Amandine Gout,Daniel Suter,Lilian Calvet,Lukas Zingg,Jennifer Straub,Nicola Alessandro Cavalcanti,Mazda Farshad,Philipp Fürnstahl,Hooman Esfandiari

2024-06-18

Abstract:This study tackles key obstacles in adopting surgical navigation in orthopedic surgeries, including time, cost, radiation, and workflow integration challenges. Recently, our work X23D showed an approach for generating 3D anatomical models of the spine from only a few intraoperative fluoroscopic images. This negates the need for conventional registration-based surgical navigation by creating a direct intraoperative 3D reconstruction of the anatomy. Despite these strides, the practical application of X23D has been limited by a domain gap between synthetic training data and real intraoperative images. In response, we devised a novel data collection protocol for a paired dataset consisting of synthetic and real fluoroscopic images from the same perspectives. Utilizing this dataset, we refined our deep learning model via transfer learning, effectively bridging the domain gap between synthetic and real X-ray data. A novel style transfer mechanism also allows us to convert real X-rays to mirror the synthetic domain, enabling our in-silico-trained X23D model to achieve high accuracy in real-world settings. Our results demonstrated that the refined model can rapidly generate accurate 3D reconstructions of the entire lumbar spine from as few as three intraoperative fluoroscopic shots. It achieved an 84% F1 score, matching the accuracy of our previous synthetic data-based research. Additionally, with a computational time of only 81.1 ms, our approach provides real-time capabilities essential for surgery integration. Through examining ideal imaging setups and view angle dependencies, we've further confirmed our system's practicality and dependability in clinical settings. Our research marks a significant step forward in intraoperative 3D reconstruction, offering enhancements to surgical planning, navigation, and robotics.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper focuses on overcoming the barriers of widespread application of surgical navigation systems in orthopedic surgery, specifically addressing issues regarding time constraints, cost, radiation concerns, and integration of the surgical process. The research team previously proposed a method called X23D, which generates a 3D anatomical model of the spine from a small number of intraoperative perspective images, thereby avoiding the need for traditional registration-based surgical navigation. However, X23D is limited in practical application due to significant domain discrepancy between synthetic training data and real intraoperative images.

To address this problem, the paper presents a new data collection protocol, creating a paired dataset that includes both synthetic and real perspective images captured from the same viewpoint. By leveraging transfer learning to optimize deep learning models, the domain gap between synthetic and real X-ray data is effectively bridged. Additionally, they introduce an innovative approach that combines style transfer and carefully curated paired dataset to transform real X-ray images into the synthetic domain, enabling high-precision performance of the X23D model trained ex vivo in real-world environments.

Experimental results demonstrate that the improved model can rapidly generate accurate 3D reconstruction of the entire lumbar spine from only three intraoperative perspective images, achieving an F1 score of 84%, which is comparable to the benchmark set by previous studies using only synthetic data. Moreover, the computational time is only 81.1 milliseconds, providing real-time capability, which is crucial for successful integration into active surgical procedures.

The paper also explores optimal imaging settings and angle dependency, further validating the practicality and reliability of the system in clinical environments. This research represents a promising advancement in intraoperative 3D reconstruction, with the potential to enhance the performance of intraoperative surgical planning, navigation, and surgical robots.