Abstract:Hepatocellular carcinoma causes intrahepatic metastasis via the trans-portal vein. Thus, appropriate mapping of portal segments is necessary for laparoscopic anatomical liver resection. However, because of the difficulty in identifying tactile sensations and the limited surgical view of laparoscopy, augmented reality (AR) has recently been utilized in laparoscopic liver surgery to identify the tumor, vessels, and portal segments. Moreover, artificial intelligence (AI) has been employed to identify landmarks in two-dimensional (2D) images because of concerns regarding the accuracy of superimposing a three-dimensional (3D) model onto a 2D laparoscopic image. In this study, we report an AR-based projection mapping method of portal segments superimposing preoperative 3D models assisted by AI in laparoscopic surgery. The liver silhouette in laparoscopic images should be detected to superimpose 3D models. Labeled liver silhouettes were obtained from 380 images in surgical videos as learning images to implement AI-based silhouette detection. To implement this technique, we used Detectron2, a PyTorch-based object detection library by Facebook AI Research (Now, Meta AI, Menlo Park, California, United States). In the videos, the liver edges were displayed as green outlines according to AI. Additionally, 3D liver models with segmental mapping were generated using the open-source software 3D Slicer from computed tomography images. For AR display, we utilized the model target function of Vuforia SDK (PTC, Inc., Boston, Massachusetts, United States), an industrial AR library with silhouette-based AR display. Lastly, we merged the AI output video with a 3D model in Unity (Unity Software Inc., San Francisco, California, United States) to establish the projection mapping of the portal segment on 2D surgical images. The accuracy was assessed by measuring the maximum error between the liver edges of laparoscopic images and 3D liver silhouettes in five surgical videos. The maximum error between liver edges and 3D model silhouettes ranged from 4 mm to 22 mm in the AI-based approach and 12 mm to 55 mm in the non-AI-based approach. Meanwhile, the mean error was 14.5 and 31.2 mm in the AI-based and non-AI-based approaches, respectively. Despite camera movement, 3D AR displays were maintained. Thus, our AI-assisted projection mapping of the portal segment could offer a new approach for laparoscopic anatomical liver resection.

Intraoperative artificial intelligence system identifying liver vessels in laparoscopic liver resection: a retrospective experimental study

A novel image-guided laparoscopic liver resection with integrated fluorescent imaging and artificial intelligence: A preliminary study.

Artificial intelligence for the recognition of key anatomical structures in laparoscopic colorectal surgery.

Towards Real-time Intrahepatic Vessel Identification in Intraoperative Ultrasound-Guided Liver Surgery

Development of a Novel Artificial Intelligence System for Laparoscopic Hepatectomy

Explainable artificial intelligence prediction-based model in laparoscopic liver surgery for segments 7 and 8: an international multicenter study

Deep learning-based automatic bleeding recognition during liver resection in laparoscopic hepatectomy

Artificial intelligence-based technology for enhancing the quality of simulation, navigation, and outcome prediction for hepatectomy

Laparoscopic Projection Mapping of the Liver Portal Segment, Based on Augmented Reality Combined With Artificial Intelligence, for Laparoscopic Anatomical Liver Resection

Real-time detection of active bleeding in laparoscopic colectomy using artificial intelligence

Artificial Intelligence in Perioperative Planning and Management of Liver Resection

Predicting Safe Liver Resection Volume for Major Hepatectomy Using Artificial Intelligence

Surgical step recognition in laparoscopic distal gastrectomy using artificial intelligence: a proof-of-concept study

Evaluating surgical expertise with AI‐based automated instrument recognition for robotic distal gastrectomy

Artificial intelligence for surgical safety during laparoscopic gastrectomy for gastric cancer: Indication of anatomical landmarks related to postoperative pancreatic fistula using deep learning

Development of a Deep Learning System for Intra-Operative Identification of Cancer Metastases

Artificial Intelligence in Liver Transplantation

Artificial intelligence automated surgical phases recognition in intraoperative videos of laparoscopic pancreatoduodenectomy

Augmented Reality and Image-Guided Robotic Liver Surgery