Deep Learning-Based Regression and Classification for Automatic Landmark Localization in Medical Images

Julia M. H. Noothout,Bob D. de Vos,Jelmer M. Wolterink,Elbrich M. Postma,Paul A. M. Smeets,Richard A. P. Takx,Tim Leiner,Max A. Viergever,Ivana Išgum

DOI: https://doi.org/10.1109/TMI.2020.3009002

2020-07-10

Abstract:In this study, we propose a fast and accurate method to automatically localize anatomical landmarks in medical images. We employ a global-to-local localization approach using fully convolutional neural networks (FCNNs). First, a global FCNN localizes multiple landmarks through the analysis of image patches, performing regression and classification simultaneously. In regression, displacement vectors pointing from the center of image patches towards landmark locations are determined. In classification, presence of landmarks of interest in the patch is established. Global landmark locations are obtained by averaging the predicted displacement vectors, where the contribution of each displacement vector is weighted by the posterior classification probability of the patch that it is pointing from. Subsequently, for each landmark localized with global localization, local analysis is performed. Specialized FCNNs refine the global landmark locations by analyzing local sub-images in a similar manner, i.e. by performing regression and classification simultaneously and combining the results. Evaluation was performed through localization of 8 anatomical landmarks in CCTA scans, 2 landmarks in olfactory MR scans, and 19 landmarks in cephalometric X-rays. We demonstrate that the method performs similarly to a second observer and is able to localize landmarks in a diverse set of medical images, differing in image modality, image dimensionality, and anatomical coverage.

Image and Video Processing,Computer Vision and Pattern Recognition,Medical Physics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the automatic and precise localization of anatomical landmarks in medical images. Specifically, the authors propose a fast and accurate method to achieve automatic anatomical landmark localization in medical images using fully convolutional neural networks (FCNNs). This method adopts a global - to - local localization strategy. First, a global FCNN is used to preliminarily localize multiple landmarks, and then a specialized FCNN is used to fine - tune the position of each landmark. During the localization process, the network simultaneously performs regression and classification tasks. It predicts the displacement vector pointing from the center of the image patch to the landmark position through regression and determines whether the image patch contains the landmark of interest through classification. The final landmark position is obtained by weighted averaging the predicted displacement vectors, where the weights are determined by the posterior classification probabilities of the image patches. In the paper, the performance of this method on medical image datasets of three different modalities is evaluated: 3D coronary CT angiography (CCTA) scans, 3D olfactory MRI scans, and 2D lateral skull X - rays. The results show that this method can localize multiple types of anatomical landmarks with an accuracy comparable to that of a second observer and can be applied in different image modalities, image dimensions, and anatomical regions.

Deep Learning-Based Regression and Classification for Automatic Landmark Localization in Medical Images

CNN-based Landmark Detection in Cardiac CTA Scans

Learn Fine-Grained Adaptive Loss for Multiple Anatomical Landmark Detection in Medical Images

Automatic Localization of Landmarks in Craniomaxillofacial CBCT Images Using a Local Attention-Based Graph Convolution Network.

Joint Detection and Localization of Multiple Anatomical Landmarks Through Learning.

Spine Landmark Localization with combining of Heatmap Regression and Direct Coordinate Regression

Spatial landmark detection and tissue registration with deep learning

Fast Multiple Landmark Localisation Using a Patch-based Iterative Network

Automatic Landmarks Correspondence Detection in Medical Images with an Application to Deformable Image Registration

FDGR-Net: Feature Decouple and Gated Recalibration Network for Medical Image Landmark Detection

Fast and Accurate Craniomaxillofacial Landmark Detection Via 3D Faster R-CNN.

DLLNet: An Attention-Based Deep Learning Method for Dental Landmark Localization on High-Resolution 3D Digital Dental Models

An Attention-Guided Deep Regression Model for Landmark Detection in Cephalograms

Landmark Localization for Cephalometric Analysis Using Multiscale Image Patch-Based Graph Convolutional Networks

Robust Multi-scale Anatomical Landmark Detection in Incomplete 3D-CT Data

Automatic localization of anatomical landmarks in head cine fluoroscopy images via deep learning

You Only Learn Once: Universal Anatomical Landmark Detection

Automatic Cephalometric Landmark Identification System Based on the Multi-Stage Convolutional Neural Networks with CBCT Combination Images

A Registration and Deep Learning Approach to Automated Landmark Detection for Geometric Morphometrics

Learning-based local-to-global landmark annotation for automatic 3D cephalometry

Automatic Detection of Craniomaxillofacial Anatomical Landmarks on CBCT Images Using 3D Mask R-CNN.