Abstract:BACKGROUND: We applied a new concept of "image overlay surgery" consisting of the integration of virtual reality (VR) and augmented reality (AR) technology, in which dynamic 3D images were superimposed on the patient's actual body surface and evaluated as a reference for surgical navigation in gastrointestinal, hepatobiliary and pancreatic surgery.METHODS: We carried out seven surgeries, including three cholecystectomies, two gastrectomies and two colectomies. A Macintosh and a DICOM workstation OsiriX were used in the operating room for image analysis. Raw data of the preoperative patient information obtained via MDCT were reconstructed to volume rendering and projected onto the patient's body surface during the surgeries. For accurate registration, OsiriX was first set to reproduce the patient body surface, and the positional coordinates of the umbilicus, left and right nipples, and the inguinal region were fixed as physiological markers on the body surface to reduce the positional error.RESULTS: The registration process was non-invasive and markerlesss, and was completed within 5 min. Image overlay navigation was helpful for 3D anatomical understanding of the surgical target in the gastrointestinal, hepatobiliary and pancreatic anatomies. The surgeon was able to minimize movement of the gaze and could utilize the image assistance without interfering with the forceps operation, reducing the gap from the VR. Unexpected organ injury could be avoided in all procedures. In biliary surgery, the projected virtual cholangiogram on the abdominal wall could advance safely with identification of the bile duct. For early gastric and colorectal cancer, the small tumors and blood vessels, which usually could not be found on the gastric serosa by laparoscopic view, were simultaneously detected on the body surface by carbon dioxide-enhanced MDCT. This provided accurate reconstructions of the tumor and involved lymph node, directly linked with optimization of the surgical procedures.CONCLUSIONS: Our non-invasive markerless registration using physiological markers on the body surface reduced logistical efforts. The image overlay technique is a useful tool when highlighting hidden structures, giving more information.

Utility of real-time image fusion technology in ultrasonography-guided fiducial marker implantation for stereotactic body radiation therapy for liver tumors

Real-time ultrasound-MRI fusion image virtual navigation for locating intraspinal tumour in a pregnant woman

Ultrasound Fusion Imaging for Improving Diagnostic and Therapeutic Strategies of Focal Liver Lesions: A Preliminary Study.

Interventional Real-Time Optical Imaging Guidance for Complete Tumor Ablation

Precise liver tumor ablation: the clinical potential of US-US overlay fusion guidance

Assessment of fiducial motion in CBCT projections of the abdominal tumor using template matching and sequential stereo triangulation

Application of imaging fusion combining contrast-enhanced ultrasound and magnetic resonance imaging in detection of hepatic cellular carcinomas undetectable by conventional ultrasound

Towards the clinical integration of an image-guided navigation system for percutaneous liver tumor ablation using freehand 2D ultrasound images

Augmented reality navigation with ultrasound-assisted point cloud registration for percutaneous ablation of liver tumors

Fusion Imaging-Guided Radiofrequency Ablation for Residual Hepatocellular Carcinoma Invisible on Ultrasound after Transcatheter Arterial Chemoembolization.

[Ultrasound Guided Ablation Therapy of Hepatic Colorectal Metastases: Initial Experience of Real Time Virtual Sonography Navigation System].

3D ultrasound guided navigation system with hybrid image fusion

The Application of Ultrasonography–Computed Tomography Fusion Navigation Technology in Complex Bone Tumor Biopsy: A Randomized Double-Blind Controlled Trial

A Comparative Study of SonoVue and Sonazoid for Contrast-Enhanced Ultrasound CT/MRI Fusion Guidance During Radiofrequency Ablation of Poorly Visualized Hepatic Malignancies: A Prospective Intra-Individual Analysis

Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery

Real-Time US-18FDG-PET/CT Image Fusion for Guidance of Thermal Ablation of 18FDG-PET-Positive Liver Metastases: The Added Value of Contrast Enhancement

Real-time Virtual Navigation System for Percutaneous Ablation of Sonographically Unidentified Hepatic Malignacies

Application of imaging fusion combining CEUS and MRI in detection of HCCs undetectable by conventional ultrasound.

Feasibility of Novel Intraoperative Navigation for Anatomical Liver Resection Using Real-Time Virtual Sonography Combined with Indocyanine Green Fluorescent Imaging Technology.

Geometric modeling of hepatic arteries in 3D ultrasound with unsupervised MRA fusion during liver interventions

3D Ultrasound System for US-guided Intervention Therapy