Augmented Reality Environments for Medical Imaging and Computer-Assisted Interventions

Hongen Liao,Cristian A. Linte,Ken Masamune,Guoyan Zheng

2013-01-01

Abstract:This book constitutes the refereed proceedings of two workshops MAIR/AE-CAI 2013, held in conjunction with MICCAI 2013, held in Nagoya, Japan, in September 2013. The 29 revised full papers presented w

What problem does this paper attempt to address?

Medical Imaging and Augmented Reality

Hongen Liao,Philip Edwards,Xiaochuan Pan,Yong Fan,Guang‐Zhong Yang

2010-01-01

Abstract:The 5th International Workshop on Medical Imaging and Augmented Reality, MIAR 2010, was held at the China National Convention Center (CNCC), B- jing, China on September 19–20, 2010. MIAR has remained
3d Medical Imaging And Augmented Reality For Image-Guided Surgery

Hongen Liao

DOI: https://doi.org/10.1007/978-1-4614-0064-6_27

2011-01-01

Abstract:Rapid advances in medical imaging, including its growing application in diagnosis, have attracted interest in the integration of research in the life sciences, medicine, chemical & physical sciences, and engineering. The goals of future biomedicine are to develop minimally invasive precision diagnostic and therapeutic techniques for human diseases by means of bio-medical imaging, bio-robotic, and precision and control engineering. This chapter introduces a variety of augmented reality techniques that were integrated into image-guided diagnostic and therapeutic system, including multi-disciplinary researches of medical imaging and augmented reality for image-guided intervention and medical implementation.
Augmented Reality Computational Spectral Imaging for Surgical Guidance

Tao Chenning,Zheng Zhenrong

DOI: https://doi.org/10.3788/LOP202259.2011014

2022-01-01

Abstract:Surgical guidance based on spectral imaging can effectively identify various tissue types by analyzing the spectrum differences of different tissues, which has a significant application value. However, the current sampling speed of spectral imaging and the presentation of spectral images severely limit its clinical applications. This paper proposes an augmented reality (AR) computational spectral imaging system for facilitating surgical guidance. High-quality spectral reconstruction is achieved in a single detection using RGB imaging devices and multispectral single-pixel detectors, which greatly improves the imaging speed and reconstruction efficiency. Principal component analysis and spectral angle mapping are adopted to extract the effective information of the spectral images, which are then used to highlight the characteristic tissue area. The image enhancement effect based on the fusion of the spectral image and actual surgical area is demonstrated in the head-mounted AR display device.
Augmented reality systems for medical applications

Son-Lik Tang,Chee-Keong Kwoh,Ming-Yeong Teo,Ng Wan Sing,Keck-Voon Ling

DOI: https://doi.org/10.1109/51.677169

1998-01-01

IEEE Engineering in Medicine and Biology Magazine

Abstract:Augmented reality (AR) is a technology in which a computer-generated image is superimposed onto the user's vision of the real world, giving the user additional information generated from the computer model. This technology is different from virtual reality, in which the user is immersed in a virtual world generated by the computer. Rather, the AR system brings the computer into the "world" of the user by augmenting the real environment with virtual objects. Using an AR system, the user's view of the real world is enhanced. This enhancement may be in the form of labels, 3D rendered models, or shaded modifications. In this article, the authors review some of the research involving AR systems, basic system configurations, image-registration approaches, and technical problems involved with AR technology. They also touch upon the requirements for an interventive AR system, which can help guide surgeons in executing a surgical plan.

English Else
A Web-Based Augmented Reality Approach to Instantly View and Display 4D Medical Images

Huy Le,Minh Nguyen,Wei Qi Yan

DOI: https://doi.org/10.1007/978-3-030-41299-9_54

2019-01-01

Abstract:In recent years, development in non-invasive and painless medical imaging such as Computed Tomography (CT) or MRI (magnetic resonance imaging), has improved the process of diseases diagnosis and clarification, including tumours, cysts, injuries, and cancers. The full-body scanner with superior spatial resolution provides essential details of complicated anatomical structures for effective diagnostics. However, it is challenging for a physician in glance over a large data-set of over hundreds or even thousands of images (2D “slices” of the body). Consider a case when a doctor wants to view a patient’s CT or MRI scans for analysing, he needs to review and compare among many layers of 2D image stacks (many 2D slices make a 3D stack). If the patient is scanned multiple time (three consecutive months, for instance) to confirm the growth of the tumours, the dataset is turned to be 4D (time-stamp added). The manual analysing process is time-consuming, troublesome and labour-intensive. The innovation of Augmented Reality (AR) in the last few decades allows us to illuminate this problem. In this paper, we propose an AR technique which assists the doctor in instantly accessing and viewing a patient’s set of medical images quickly and easily. The doctor can use an optical head-mounted display such as the Google Glass, or a VR Headset such as the Samsung Gear VR, or a general smartphone such as the Apple iPhone X. He looks at one palm-sized AR Tag with patient’s document embedded with a QR code, and the smart device could detect and download the patient’s data using the decrypted QR code and display layers of CT or MRI images right on top of the AR tag. Looking in and out from the tag allows the doctor to see the above or below of the current viewing layer. Moreover, shifting the looking orientation left or right allows the doctor to see the same layer of images but in different timestamp (e.g. previous or next monthly scans). Our obtained results demonstrated that this technique enhances the diagnosing process, save cost and time for medical practice.
Enhancing Visualisation of Anatomical Presentation and Education Using Marker-based Augmented Reality Technology on Web-based Platform

I. Wang,M. Nguyen,H. Le,W. Yan,S. Hooper

DOI: https://doi.org/10.1109/avss.2018.8639147

2018-01-01

Abstract:The domain of teaching medicine involves the mastery of many complex skills that almost always need to be performed in real life situations following very high professional standards. However, this training is not always possible for various reasons such as ethics, safety and costs. Virtual reality (VR) and augmented reality (AR) have started to be widely used as alternative medical teaching practices. Our proposed AR system works online, so no installation is required on a users device; it only requires a generic colour web-cam to track a pictorial AR tag which contains a hidden QR code. These QR codes contain data, such as the ID of a three-dimensional (3D) model, and merge it with text so that it can be used as both an identity and tag pattern in the AR marker. The system can then show the corresponding computer-generated 3D anatomical models of organs; relevant text information about the subject is displayed above the AR Tag. The tag is numerically encrypted and decrypted and detectable by shape and orientation. Different from other similar techniques, our AR Tag is both a bar-code and a template marker, QR code is used to load a previously setup website, and then the detail of that QR code is used as a template to identify the border and orientation of the marker. The system is thus faster and more robust that allows users to control and navigate the 3D environment by zooming in and out and rotating left and right. It is hoped that this virtual environment will help reduce the need for real-life surgical practice, instead of increasing intuition, the direct 3D perception of the human body and other 3D medical imaging data (mimesis). This system could even be further developed to present the framework of a patient’s anatomy.
3D augmented reality based orthopaedic interventions

Xinran Zhang,Zhencheng Fan,Hongen Liao,Junchen Wang

DOI: https://doi.org/10.1007/978-3-319-23482-3_4

2016-01-01

Abstract:Augmented reality (AR) techniques, which can merge virtual computer-generated guidance information into real medical interventions, help surgeons obtain dynamic "see-through" scenes during orthopaedic interventions. Among various AR techniques, 3D integral videography (IV) image overlay is a promising solution because of its simplicity in implementation as well as the ability to produce a full parallax augmented natural view for multiple observers and improve surgeons' hand-eye coordination. To obtain a precise fused result, patient-3D image registration is a vital technique in the IV overlay based orthopaedic interventions. Marker or marker-less based registration techniques are alternative depending on a particular clinical application. According to accurate AR information, minimally invasive therapy including cutting, drilling, implantation and other related operations, can be performed more easily and safely. This chapter reviews related augmented reality techniques for image-guided surgery and analyses several examples about clinical applications. Eventually, we discuss the future development of 3D AR based orthopaedic interventions.
Visualization, Registration and Tracking Techniques for Augmented Reality Guided Surgery: a Review

Longfei Ma,Tianqi Huang,Jie Wang,Hongen Liao

DOI: https://doi.org/10.1088/1361-6560/acaf23

2023-01-01

Abstract:Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories of in situ visualization and non in situ visualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.
Multi-sensor Spatial Augmented Reality for Visualizing the Invisible Thermal Information of 3D Objects

Yanpeng Cao,Yafei Dong,Fan Wang,Jiangxin Yang,Yanlong Cao,Xin Li

DOI: https://doi.org/10.1016/j.optlaseng.2021.106634

IF: 5.666

2021-01-01

Optics and Lasers in Engineering

Abstract:In recent years, infrared imaging technology has been widely used to capture valuable thermal information of target objects for various applications such as construction engineering, advanced manufacturing, and medical diagnosis. However, it still remains a challenging task to visualize the invisible thermal information of three-dimensional objects for further evaluation and decision making. In this paper, we build a multi-sensor system consisting of a long-wave infrared camera, a RGBD camera, and a digital projector for multimodal data acquisition, real-time 3D thermographic reconstruction, and projector-based spatial augmented reality. We propose a hybrid method that combines the benefits of model-based ray casting and frame-based image warping to generate high-quality projected images with complete view coverage and abundant thermal details for visualizing the invisible thermal information on 3D objects. The effectiveness of the proposed method has been validated for two typical thermal imaging applications including human body temperature monitoring and non-destructive evaluation of composite materials.
Real-time Geometry-Aware Augmented Reality in Minimally Invasive Surgery

Long Chen,Wen Tang,Nigel W. John

DOI: https://doi.org/10.48550/arXiv.1708.01234

2017-08-04

Abstract:The potential of Augmented Reality (AR) technology to assist minimally invasive surgeries (MIS) lies in its computational performance and accuracy in dealing with challenging MIS scenes. Even with the latest hardware and software technologies, achieving both real-time and accurate augmented information overlay in MIS is still a formidable task. In this paper, we present a novel real-time AR framework for MIS that achieves interactive geometric aware augmented reality in endoscopic surgery with stereo views. Our framework tracks the movement of the endoscopic camera and simultaneously reconstructs a dense geometric mesh of the MIS scene. The movement of the camera is predicted by minimising the re-projection error to achieve a fast tracking performance, while the 3D mesh is incrementally built by a dense zero mean normalised cross correlation stereo matching method to improve the accuracy of the surface reconstruction. Our proposed system does not require any prior template or pre-operative scan and can infer the geometric information intra-operatively in real-time. With the geometric information available, our proposed AR framework is able to interactively add annotations, localisation of tumours and vessels, and measurement labelling with greater precision and accuracy compared with the state of the art approaches.

Computer Vision and Pattern Recognition
Personalized Augmented Reality for Anatomy Education

Meng Ma,Pascal Fallavollita,Ina Seelbach,Anna Maria Von der Heide,Ekkehard Euler,Jens Waschke,Nassir Navab

DOI: https://doi.org/10.1002/ca.22675

2015-01-01

Clinical Anatomy

Abstract:Anatomy education is a challenging but vital element in forming future medical professionals. In this work, a personalized and interactive augmented reality system is developed to facilitate education. This system behaves as a “magic mirror” which allows personalized in‐situ visualization of anatomy on the user's body. Real‐time volume visualization of a CT dataset creates the illusion that the user can look inside their body. The system comprises a RGB‐D sensor as a real‐time tracking device to detect the user moving in front of a display. In addition, the magic mirror system shows text information, medical images, and 3D models of organs that the user can interact with. Through the participation of 7 clinicians and 72 students, two user studies were designed to respectively assess the precision and acceptability of the magic mirror system for education. The results of the first study demonstrated that the average precision of the augmented reality overlay on the user body was 0.96 cm, while the results of the second study indicate 86.1% approval for the educational value of the magic mirror, and 91.7% approval for the augmented reality capability of displaying organs in three dimensions. The usefulness of this unique type of personalized augmented reality technology has been demonstrated in this paper. Clin. Anat. 29:446–453, 2016. © 2015 Wiley Periodicals, Inc.
2 Conventional AR Techniques and Medical Applications 2 . 1 Slice Image Overlay Based AR Windows for Image-Guided Surgery

H. Liao

2011-01-01

Abstract:Rapid advances in medical imaging, including its growing application in diagnosis, have attracted interest in the integration of research in the life sciences, medicine, chemical & physical sciences, and engineering. The goals of future biomedicine are to develop minimally invasive precision diagnostic and therapeutic techniques for human diseases by means of bio-medical imaging, bio-robotic, and precision and control engineering. This chapter introduces a variety of augmented reality techniques that were integrated into image-guided diagnostic and therapeutic system, including multi-disciplinary researches of medical imaging and augmented reality for image-guided intervention and medical implementation.
Spatiotemporal-Aware Augmented Reality: Redefining HCI in Image-Guided Therapy

Javad Fotouhi,Arian Mehrfard,Tianyu Song,Alex Johnson,Greg Osgood,Mathias Unberath,Mehran Armand,Nassir Navab

DOI: https://doi.org/10.48550/arXiv.2003.02260

2020-03-05

Abstract:Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. As a consequence, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. The main contribution of this paper is to reveal how exemplary workflows are redefined by taking full advantage of head-mounted displays when entirely co-registered with the imaging system at all times. The proposed AR landscape is enabled by co-localizing the users and the imaging devices via the operating room environment and exploiting all involved frustums to move spatial information between different bodies. The awareness of the system from the geometric and physical characteristics of X-ray imaging allows the redefinition of different human-machine interfaces. We demonstrate that this AR paradigm is generic, and can benefit a wide variety of procedures. Our system achieved an error of $4.76\pm2.91$ mm for placing K-wire in a fracture management procedure, and yielded errors of $1.57\pm1.16^\circ$ and $1.46\pm1.00^\circ$ in the abduction and anteversion angles, respectively, for total hip arthroplasty. We hope that our holistic approach towards improving the interface of surgery not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications and provide novel approaches of documenting procedures for training purposes.

Computer Vision and Pattern Recognition,Human-Computer Interaction,Image and Video Processing
Medical Augmented Reality: Definition, Principle Components, Domain Modeling, and Design-Development-Validation Process

Nassir Navab,Alejandro Martin-Gomez,Matthias Seibold,Michael Sommersperger,Tianyu Song,Alexander Winkler,Kevin Yu,Ulrich Eck

DOI: https://doi.org/10.3390/jimaging9010004

2022-12-23

Abstract:Three decades after the first set of work on Medical Augmented Reality (MAR) was presented to the international community, and ten years after the deployment of the first MAR solutions into operating rooms, its exact definition, basic components, systematic design, and validation still lack a detailed discussion. This paper defines the basic components of any Augmented Reality (AR) solution and extends them to exemplary Medical Augmented Reality Systems (MARS). We use some of the original MARS applications developed at the Chair for Computer Aided Medical Procedures and deployed into medical schools for teaching anatomy and into operating rooms for telemedicine and surgical guidance throughout the last decades to identify the corresponding basic components. In this regard, the paper is not discussing all past or existing solutions but only aims at defining the principle components and discussing the particular domain modeling for MAR and its design-development-validation process, and providing exemplary cases through the past in-house developments of such solutions.
Current Status and Challenge for Implementation in Clinical Application of Augmented Reality in Orthopedics Department at a Medical Center

Hsiao-Hui Chiu,Pi-Ching Wei,Man-Ling Lin,Yi-Chun Chen,Shu-Yuan Chou,Yu-Cheng Yao,Ying-Ying Yang

DOI: https://doi.org/10.20849/ijsn.v6i3.923

2021-09-22

International Journal of Studies in Nursing

Abstract:With technical development, artificial intelligence (AI) has been actively involving in the healthcare industry. Augmented reality (AR) is an interactive experience with the combination of virtual objects and a real-world environment, and the objects reside in the environment of the real world through computer-generated images for the purpose to enhance perceptual effects, which can be applied in the fields of medical education and clinical practice. Researchers have found that learning motives and interests may be raised by AR. At a medical center, the inter-specialty team from the Teaching Department and Rehabilitation Department jointly developed an AR Medical Education App, which involves 44 muscle strength and walking exercises, including 6 upper limb movements, 28 lower limb exercises, and 10 cardiorespiratory exercises. Various exercise packages can be designed by health caregivers based on patient’s needs in exercise. The Orthopedics Ward applied it in the respiration training for patients who underwent spinal surgery, preventing respiratory comorbidities. The improved postoperative pulmonary function has been found when compared with that before surgery, with statistical significance. Respiration, upper and lower limbs exercises were persistently performed in patients who underwent spinal surgery, which also has been incorporated in routine nursing care to add diverse options of patient education materials for nursing personnel. In the future, aspects that required further efforts include: expanded software equipment, simplified operation mode, increased user-friendly features for equipment, and expended plentifulness of contents.
8.2: Anatomy Education Method Using Autostereoscopic 3D Image Overlay and Mid‐Air Augmented Reality Interaction

Tianqi Huang,Nan Zhang,Xinran Zhang,Hongen Liao

DOI: https://doi.org/10.1002/sdtp.15857

2022-01-01

Abstract:The augmented reality (AR) medical education systems offer unique advantages for an interactive and hands‐on integration with the student in surgical skill training. However, there are still major differences in visualization and interaction between real surgical environments and AR‐based medical education systems. In this paper, we propose an anatomy education method using an autostereoscopic 3D image overlay and mid‐air augmented reality interaction with soft tissue deformation. To achieve good depth perception and intuitive interaction, we introduce a model‐based occlusion handling for accurate occlusion processing in real time and a position‐based dynamics (PBD) simulation method for virtual soft tissue deformation in real time. The experimental results showed that our approach could provide 3D visualization and interaction of virtual soft tissues and handle the mutual occlusion automatically in real time. Our proposed medical education system will provide users with a visualization and interaction experience close to real surgical environments.
Proceedings VRCAI 2013: 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry,The Chinese University of Hong Kong, Hong Kong November 17–19, 2013

Charlie C. L. Wang,Nadia Magnenat Thalmann,Zhigeng Pan,Xiaogang Jin,Caterina Rizzi,Xiaopeng Zhang

2013-01-01

Abstract:Department of Mechanical and Automation Engineering (MAE), The Chinese University of Hong Kong (CUHK) hosted the 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry (VRCAI 2013), from 17-19 November 2013 on the campus of CUHK. Virtual Reality Continuum emphasizes on the consistence of virtual world and the real world. Spanning across next-generation info-communication environments like Virtual Reality (VR), Augmented Virtuality (AV), Augmented Reality (AR) and Mixed Reality (MR), Virtual Reality Continuum is key in the way we define and interact, with and within, the virtual world. To advance research in the Virtual Reality Continuum fields, VRCAI 2013 provides a forum for scientists, researchers, developers, users and industry leaders in the international Virtual Reality Continuum community to come together to share experiences, exchange ideas and spur one another in the knowledge of this fast-growing arena.
3D Visualization and Augmented Reality for Orthopedics.

Longfei Ma,Zhencheng Fan,Guochen Ning,Xinran Zhang,Hongen Liao

DOI: https://doi.org/10.1007/978-981-13-1396-7_16

2018-01-01

Abstract:Augmented reality (AR) techniques play an important role in the field of minimally invasive surgery for orthopedics. AR can improve the hand-eye coordination by providing surgeons with the merged surgical scene, which enables surgeons to perform surgical operations more easily. To display the navigation information in the AR scene, medical image processing and three-dimensional (3D) visualization of the important anatomical structures are required. As a promising 3D display technique, integral videography (IV) can produce an autostereoscopic image with full parallax and continuous viewing points. Moreover, IV-based 3D AR navigation technique is proposed to present intuitive scene and has been applied in orthopedics, including oral surgery and spine surgery. The accurate patient-image registration, as well as the real-time target tracking for surgical tools and the patient, can be achieved. This paper overviews IV-based AR navigation and the applications in orthopedics, discusses the infrastructure required for successful implementation of IV-based approaches, and outlines the challenges that must be overcome for IV-based AR navigation to advance further development.
Feasibility Study of Using Augmented Mirrors for Alignment Task during Orthopaedic Procedures in Mixed Reality

Xiaorui Zhang,Andreas Keller,Mehran Armand,Alejandro Martin Gomez

DOI: https://doi.org/10.1109/ismar-adjunct60411.2023.00139

Abstract:Accurate depth estimation poses a significant challenge in egocentric Augmented Reality (AR), particularly for precision-dependent tasks in the medical field, such as needle or tool insertions during percutaneous procedures. Augmented Mirrors (AMs) provide a unique solution to this problem by offering additional non-egocentric viewpoints that enhance spatial understanding of an AR scene. Despite the perceptual advantages of using AMs, their practical utility has yet to be thoroughly tested. In this work, we present results from a pilot study involving five participants tasked with simulating epidural injection procedures in an AR environment, both with and without the aid of an AM. Our findings indicate that using AM contributes to reducing mental effort while improving alignment accuracy. These results highlight the potential of AM as a powerful tool for AR-enabled medical procedures, setting the stage for future exploration involving medical professionals.

Augmented Reality Environments for Medical Imaging and Computer-Assisted Interventions

Medical Imaging and Augmented Reality

3d Medical Imaging And Augmented Reality For Image-Guided Surgery

Augmented Reality Computational Spectral Imaging for Surgical Guidance

Augmented reality systems for medical applications

A Web-Based Augmented Reality Approach to Instantly View and Display 4D Medical Images

Enhancing Visualisation of Anatomical Presentation and Education Using Marker-based Augmented Reality Technology on Web-based Platform

3D augmented reality based orthopaedic interventions

Visualization, Registration and Tracking Techniques for Augmented Reality Guided Surgery: a Review

Multi-sensor Spatial Augmented Reality for Visualizing the Invisible Thermal Information of 3D Objects

Real-time Geometry-Aware Augmented Reality in Minimally Invasive Surgery

Personalized Augmented Reality for Anatomy Education

2 Conventional AR Techniques and Medical Applications 2 . 1 Slice Image Overlay Based AR Windows for Image-Guided Surgery

Spatiotemporal-Aware Augmented Reality: Redefining HCI in Image-Guided Therapy

Medical Augmented Reality: Definition, Principle Components, Domain Modeling, and Design-Development-Validation Process

Current Status and Challenge for Implementation in Clinical Application of Augmented Reality in Orthopedics Department at a Medical Center

8.2: Anatomy Education Method Using Autostereoscopic 3D Image Overlay and Mid‐Air Augmented Reality Interaction

Proceedings VRCAI 2013: 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry,The Chinese University of Hong Kong, Hong Kong November 17–19, 2013

3D Visualization and Augmented Reality for Orthopedics.

Feasibility Study of Using Augmented Mirrors for Alignment Task during Orthopaedic Procedures in Mixed Reality