On a simple definition of computable function of a real variable-with applications to functions of a complex variable

M. B. Pour-El,Jerome Caldwell

DOI: https://doi.org/10.1002/malq.19750210102

Mathematical Logic Quarterly

Abstract:

What problem does this paper attempt to address?

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Yao Zhang,Nanjun He,Jiawei Yang,Yuexiang Li,Dong Wei,Yawen Huang,Yang Zhang,Zhiqiang He,Yefeng Zheng

DOI: https://doi.org/10.1007/978-3-031-16443-9_11

2022-01-01

Abstract:Accurate brain tumor segmentation from Magnetic Resonance Imaging (MRI) is desirable to joint learning of multimodal images. However, in clinical practice, it is not always possible to acquire a complete set of MRIs, and the problem of missing modalities causes severe performance degradation in existing multimodal segmentation methods. In this work, we present the first attempt to exploit the Transformer for multimodal brain tumor segmentation that is robust to any combinatorial subset of available modalities. Concretely, we propose a novel multimodal Medical Transformer (mmFormer) for incomplete multimodal learning with three main components: the hybrid modality-specific encoders that bridge a convolutional encoder and an intra-modal Transformer for both local and global context modeling within each modality; an inter-modal Transformer to build and align the long-range correlations across modalities for modality-invariant features with global semantics corresponding to tumor region; a decoder that performs a progressive up-sampling and fusion with the modality-invariant features to generate robust segmentation. Besides, auxiliary regularizers are introduced in both encoder and decoder to further enhance the model's robustness to incomplete modalities. We conduct extensive experiments on the public BraTS $2018$ dataset for brain tumor segmentation. The results demonstrate that the proposed mmFormer outperforms the state-of-the-art methods for incomplete multimodal brain tumor segmentation on almost all subsets of incomplete modalities, especially by an average 19.07% improvement of Dice on tumor segmentation with only one available modality. The code is available at https://github.com/YaoZhang93/mmFormer.
Accurate Whole-Brain Segmentation for Bimodal PET/MR Images Via a Cross-Attention Mechanism

Wenbo Li,Zhenxing Huang,Qiyang Zhang,Na Zhang,Wenjie Zhao,Yaping Wu,Jianmin Yuan,Yang,Yan Zhang,Yongfeng Yang,Hairong Zheng,Dong Liang,Meiyun Wang,Zhanli Hu

DOI: https://doi.org/10.1109/trpms.2024.3413862

2024-01-01

IEEE Transactions on Radiation and Plasma Medical Sciences

Abstract:The PET/MRI system plays a significant role in the functional and anatomical quantification of the brain, providing accurate diagnostic data for a variety of brain disorders. However, most of the current methods for segmenting the brain are based on unimodal MRI and rarely combine structural and functional dual-modality information. Therefore, we aimed to employ deep learning techniques to achieve automatic and accurate segmentation of the whole brain while incorporating functional and anatomical information. To leverage dual-modality information, a novel 3D network with a cross-attention module was proposed to capture the correlation between dual-modality features and improve segmentation accuracy. Moreover, several deep learning methods were employed as comparison measures to evaluate the model performance, with the Dice similarity coefficient (DSC), Jaccard index (JAC), recall, and precision serving as quantitative metrics. Experimental results demonstrated our advantages in whole-brain segmentation, achieving an 85.35% DSC, 77.22% JAC, 88.86% recall, and 84.81% precision, which were better than those comparative methods. In addition, consistent and correlated analyses based on segmentation results also demonstrated that our approach achieved superior performance. In future work, we will try to apply our method to other multimodal tasks, such as PET/CT data analysis.
Deep learning-based 3D brain multimodal medical image registration

Liwei Deng,Qi Lan,Qiang Zhi,Sijuan Huang,Jing Wang,Xin Yang

DOI: https://doi.org/10.1007/s11517-023-02941-9

Abstract:Medical image registration is a critical preprocessing step in medical image analysis. While traditional medical image registration techniques have matured, their registration speed and accuracy still fall short of clinical requirements. In this paper, we propose an improved VoxelMorph network incorporating ResNet modules and CBAM (RCV-Net), for 3D multimodal unsupervised registration. Unlike popular convolution-based U-shaped registration networks like VoxelMorph, RCV-Net incorporates the convolutional block attention module (CBAM) during the convolution process. This inclusion enhances the feature map information extraction capabilities during training and effectively prevents information loss. Additionally, we introduce a lightweight and residual network module at the network's base, which enhances learning ability without significantly increasing training parameters. To evaluate the superiority of our registration model, we utilize evaluation metrics such as structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and mean square error (MSE). Experimental results demonstrate that our proposed network structure outperforms current state-of-the-art methods, yielding better performance in multimodal registration tasks. Furthermore, generalization testing on databases outside of the training set has confirmed the optimal registration effectiveness of our model.
Cross-Modality Coherent Registration of Whole Mouse Brains

Hanchuan Peng,Lei Qu,Yuanyuan Li,Peng Xie,Lijuan Liu,Yimin Wang,Jun Wu,Yu Liu,Tao Wang,Longfei Li,Kaixuan Guo,Wan Wan,Lei Ouyang,Feng Xiong,Yefeng Zheng,Fang Xu,Hui Gong,Hongwei Dong,Qingming Luo,Guo-Qiang Bi,Michael Hawrylycz,Hongkui Zeng

DOI: https://doi.org/10.21203/rs.3.rs-321118/v1

2021-01-01

Abstract:Abstract Recent whole brain mapping projects are collecting large-scale 3D images using powerful and informative modalities, such as STPT, fMOST, VISoR, or MRI. Registration of these multi-dimensional whole-brain images onto a standard atlas is essential for characterizing neuron types and constructing brain wiring diagrams. However, cross-modality image registration is challenging due to intrinsic variations of brain anatomy and artifacts resulted from different sample preparation methods and imaging modalities. We introduced a cross-modality registration method, called mBrainAligner, which uses coherent landmark mapping as well as deep neural networks to align whole mouse brain images to the standard Allen Common Coordinate Framework atlas. We also built a single cell resolution atlas using the fMOST modality, and used our method to generate whole brain map of 3D full single neuron morphology and neuron cell types.
A hybrid deep learning framework for integrated segmentation and registration: evaluation on longitudinal white matter tract changes

Bo Li,Wiro Niessen,Stefan Klein,Marius de Groot,Arfan Ikram,Meike Vernooij,Esther Bron

DOI: https://doi.org/10.48550/arXiv.1908.10221

2019-08-26

Abstract:To accurately analyze changes of anatomical structures in longitudinal imaging studies, consistent segmentation across multiple time-points is required. Existing solutions often involve independent registration and segmentation components. Registration between time-points is used either as a prior for segmentation in a subsequent time point or to perform segmentation in a common space. In this work, we propose a novel hybrid convolutional neural network (CNN) that integrates segmentation and registration into a single procedure. We hypothesize that the joint optimization leads to increased performance on both tasks. The hybrid CNN is trained by minimizing an integrated loss function composed of four different terms, measuring segmentation accuracy, similarity between registered images, deformation field smoothness, and segmentation consistency. We applied this method to the segmentation of white matter tracts, describing functionally grouped axonal fibers, using N=8045 longitudinal brain MRI data of 3249 individuals. The proposed method was compared with two multistage pipelines using two existing segmentation methods combined with a conventional deformable registration algorithm. In addition, we assessed the added value of the joint optimization for segmentation and registration separately. The hybrid CNN yielded significantly higher accuracy, consistency and reproducibility of segmentation than the multistage pipelines, and was orders of magnitude faster. Therefore, we expect it can serve as a novel tool to support clinical and epidemiological analyses on understanding microstructural brain changes over time.

Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning
Cross-modal coherent registration of whole mouse brains

Lei Qu,Yuanyuan Li,Peng Xie,Lijuan Liu,Yimin Wang,Jun Wu,Yu Liu,Tao Wang,Longfei Li,Kaixuan Guo,Wan Wan,Lei Ouyang,Feng Xiong,Anna C. Kolstad,Zhuhao Wu,Fang Xu,Yefeng Zheng,Hui Gong,Qingming Luo,Guoqiang Bi,Hongwei Dong,Michael Hawrylycz,Hongkui Zeng,Hanchuan Peng

DOI: https://doi.org/10.1038/s41592-021-01334-w

IF: 48

2021-12-09

Nature Methods

Abstract:Recent whole-brain mapping projects are collecting large-scale three-dimensional images using modalities such as serial two-photon tomography, fluorescence micro-optical sectioning tomography, light-sheet fluorescence microscopy, volumetric imaging with synchronous on-the-fly scan and readout or magnetic resonance imaging. Registration of these multi-dimensional whole-brain images onto a standard atlas is essential for characterizing neuron types and constructing brain wiring diagrams. However, cross-modal image registration is challenging due to intrinsic variations of brain anatomy and artifacts resulting from different sample preparation methods and imaging modalities. We introduce a cross-modal registration method, mBrainAligner, which uses coherent landmark mapping and deep neural networks to align whole mouse brain images to the standard Allen Common Coordinate Framework atlas. We build a brain atlas for the fluorescence micro-optical sectioning tomography modality to facilitate single-cell mapping, and used our method to generate a whole-brain map of three-dimensional single-neuron morphology and neuron cell types.

biochemical research methods
Unsupervised deep learning registration model for multimodal brain images

Samaneh Abbasi,Alireza Mehdizadeh,Hamid Reza Boveiri,Mohammad Amin Mosleh Shirazi,Reza Javidan,Raouf Khayami,Meysam Tavakoli

DOI: https://doi.org/10.1002/acm2.14177

2023-10-14

Journal of Applied Clinical Medical Physics

Abstract:Multimodal image registration is a key for many clinical image‐guided interventions. However, it is a challenging task because of complicated and unknown relationships between different modalities. Currently, deep supervised learning is the state‐of‐theart method at which the registration is conducted in end‐to‐end manner and one‐shot. Therefore, a huge ground‐truth data is required to improve the results of deep neural networks for registration. Moreover, supervised methods may yield models that bias towards annotated structures. Here, to deal with above challenges, an alternative approach is using unsupervised learning models. In this study, we have designed a novel deep unsupervised Convolutional Neural Network (CNN)‐based model based on computer tomography/magnetic resonance (CT/MR) co‐registration of brain images in an affine manner. For this purpose, we created a dataset consisting of 1100 pairs of CT/MR slices from the brain of 110 neuropsychic patients with/without tumor. At the next step, 12 landmarks were selected by a well‐experienced radiologist and annotated on each slice resulting in the computation of series of metrics evaluation, target registration error (TRE), Dice similarity, Hausdorff, and Jaccard coefficients. The proposed method could register the multimodal images with TRE 9.89, Dice similarity 0.79, Hausdorff 7.15, and Jaccard 0.75 that are appreciable for clinical applications. Moreover, the approach registered the images in an acceptable time 203 ms and can be appreciable for clinical usage due to the short registration time and high accuracy. Here, the results illustrated that our proposed method achieved competitive performance against other related approaches from both reasonable computation time and the metrics evaluation.

radiology, nuclear medicine & medical imaging
Deep learning facilitates fully automated brain image registration of optoacoustic tomography and magnetic resonance imaging

Yexing Hu,Berkan Lafci,Artur Luzgin,Hao Wang,Jan Klohs,Xose Luis Dean-Ben,Ruiqing Ni,Daniel Razansky,Wuwei Ren

DOI: https://doi.org/10.1364/BOE.458182

2022-08-18

Abstract:Multispectral optoacoustic tomography (MSOT) is an emerging optical imaging method providing multiplex molecular and functional information from the rodent brain. It can be greatly augmented by magnetic resonance imaging (MRI) which offers excellent soft-tissue contrast and high-resolution brain anatomy. Nevertheless, registration of MSOT-MRI images remains challenging, chiefly due to the entirely different image contrast rendered by these two modalities. Previously reported registration algorithms mostly relied on manual user-dependent brain segmentation, which compromised data interpretation and quantification. Here we propose a fully automated registration method for MSOT-MRI multimodal imaging empowered by deep learning. The automated workflow includes neural network-based image segmentation to generate suitable masks, which are subsequently registered using an additional neural network. The performance of the algorithm is showcased with datasets acquired by cross-sectional MSOT and high-field MRI preclinical scanners. The automated registration method is further validated with manual and half-automated registration, demonstrating its robustness and accuracy.
A novel brain registration model combining structural and functional MRI information

Baolong Li,Yuhu Shi,Lei Wang,Weiming Zeng,Changming Zhu

2024-09-26

Abstract:Although developed functional magnetic resonance imaging (fMRI) registration algorithms based on deep learning have achieved a certain degree of alignment of functional area, they underutilized fine structural information. In this paper, we propose a semi-supervised convolutional neural network (CNN) registration model that integrates both structural and functional MRI information. The model first learns to generate deformation fields by inputting structural MRI (T1w-MRI) into the CNN to capture fine structural information. Then, we construct a local functional connectivity pattern to describe the local fMRI information, and use the Bhattacharyya coefficient to measure the similarity between two fMRI images, which is used as a loss function to facilitate the alignment of functional areas. In the inter-subject registration experiment, our model achieved an average number of voxels exceeding the threshold of 4.24 is 2248 in the group-level t-test maps for the four functional brain networks (default mode network, visual network, central executive network, and sensorimotor network). Additionally, the atlas-based registration experiment results show that the average number of voxels exceeding this threshold is 3620. The results are the largest among all methods. Our model achieves an excellent registration performance in fMRI and improves the consistency of functional regions. The proposed model has the potential to optimize fMRI image processing and analysis, facilitating the development of fMRI applications.

Image and Video Processing
Multimodal Brain Tumor Segmentation Boosted by Monomodal Normal Brain Images

Huabing Liu,Zhengze Ni,Dong Nie,Dinggang Shen,Jinda Wang,Zhenyu Tang

DOI: https://doi.org/10.1109/tip.2024.3359815

IF: 10.6

2024-02-14

IEEE Transactions on Image Processing

Abstract:Many deep learning based methods have been proposed for brain tumor segmentation. Most studies focus on deep network internal structure to improve the segmentation accuracy, while valuable external information, such as normal brain appearance, is often ignored. Inspired by the fact that radiologists often screen lesion regions with normal appearance as reference in mind, in this paper, we propose a novel deep framework for brain tumor segmentation, where normal brain images are adopted as reference to compare with tumor brain images in a learned feature space. In this way, features at tumor regions, i.e., tumor-related features, can be highlighted and enhanced for accurate tumor segmentation. It is known that routine tumor brain images are multimodal, while normal brain images are often monomodal. This causes the feature comparison a big issue, i.e., multimodal vs. monomodal. To this end, we present a new feature alignment module (FAM) to make the feature distribution of monomodal normal brain images consistent/inconsistent with multimodal tumor brain images at normal/tumor regions, making the feature comparison effective. Both public (BraTS2022) and in-house tumor brain image datasets are used to evaluate our framework. Experimental results demonstrate that for both datasets, our framework can effectively improve the segmentation accuracy and outperforms the state-of-the-art segmentation methods. Codes are available at https://github.com/hb-liu/Normal-Brain-Boost-Tumor-Segmentation.

computer science, artificial intelligence,engineering, electrical & electronic
Metabolic fate of 6,6,9-trimethyl-9-azabicyclo(3,3,1)non-3 -yl , -di(2-thienyl)glycolate hydrochloride monohydrate (PG-501).

T. Meshi,S. Nakamura,Y. Sato

DOI: https://doi.org/10.1248/CPB.20.1687

1972-08-25

Chemical and Pharmaceutical Bulletin

Abstract:
A Context-Supported Deep Learning Framework for Multimodal Brain Imaging Classification

Jianmin Jiang,Ahmed Fares,Sheng-Hua Zhong

DOI: https://doi.org/10.1109/thms.2019.2904615

2019-12-01

Abstract:Over the past decade, content-based multimedia systems have realized success. By comparison, brain imaging and classification systems demand more efforts for improvement with respect to accuracy, generalization, and interpretation. The relationship between electroencephalogram (EEG) signals and corresponding multimedia content needs to be further explored. In this paper, we integrate implicit and explicit learning modalities into a context-supported deep learning framework. We propose an improved solution for the task of brain imaging classification via EEG signals. In our proposed framework, we introduce a consistency test by exploiting the context of brain images and establishing a mapping between visual-level features and cognitive-level features inferred based on EEG signals. In this way, a multimodal approach can be developed to deliver an improved solution for brain imaging and its classification based on explicit learning modalities and research from the image processing community. In addition, a number of fusion techniques are investigated in this work to optimize individual classification results. Extensive experiments have been carried out, and their results demonstrate the effectiveness of our proposed framework. In comparison with the existing state-of-the-art approaches, our proposed framework achieves superior performance in terms of not only the standard visual object classification criteria, but also the exploitation of transfer learning. For the convenience of research dissemination, we make the source code publicly available for downloading at GitHub (https://github.com/aneeg/dual-modal-learning).
The Method of Multimodal MRI Brain Image Segmentation Based on Differential Geometric Features

Yongpei Zhu,Zicong Zhou,Guojun Liao,Qianxi Yang,Kehong Yuan

DOI: https://doi.org/10.48550/arXiv.1811.04281

2019-03-06

Abstract:Accurate segmentation of brain tissue in magnetic resonance images (MRI) is a diffcult task due to different types of brain abnormalities. Using information and features from multimodal MRI including T1, T1-weighted inversion recovery (T1-IR) and T2-FLAIR and differential geometric features including the Jacobian determinant(JD) and the curl vector(CV) derived from T1 modality can result in a more accurate analysis of brain images. In this paper, we use the differential geometric information including JD and CV as image characteristics to measure the differences between different MRI images, which represent local size changes and local rotations of the brain image, and we can use them as one CNN channel with other three modalities (T1-weighted, T1-IR and T2-FLAIR) to get more accurate results of brain segmentation. We test this method on two datasets including IBSR dataset and MRBrainS datasets based on the deep voxelwise residual network, namely VoxResNet, and obtain excellent improvement over single modality or three modalities and increases average DSC(Cerebrospinal Fluid (CSF), Gray Matter (GM) and White Matter (WM)) by about 1.5% on the well-known MRBrainS18 dataset and about 2.5% on the IBSR dataset. Moreover, we discuss that one modality combined with its JD or CV information can replace the segmentation effect of three modalities, which can provide medical conveniences for doctor to diagnose because only to extract T1-modality MRI image of patients. Finally, we also compare the segmentation performance of our method in two networks, VoxResNet and U-Net network. The results show VoxResNet has a better performance than U-Net network with our method in brain MRI segmentation. We believe the proposed method can advance the performance in brain segmentation and clinical diagnosis.

Computer Vision and Pattern Recognition
Deep Representation Learning For Multimodal Brain Networks

Wen Zhang,Liang Zhan,Paul Thompson,Yalin Wang

DOI: https://doi.org/10.48550/arXiv.2007.09777

2020-07-19

Computer Vision and Pattern Recognition

Abstract:Applying network science approaches to investigate the functions and anatomy of the human brain is prevalent in modern medical imaging analysis. Due to the complex network topology, for an individual brain, mining a discriminative network representation from the multimodal brain networks is non-trivial. The recent success of deep learning techniques on graph-structured data suggests a new way to model the non-linear cross-modality relationship. However, current deep brain network methods either ignore the intrinsic graph topology or require a network basis shared within a group. To address these challenges, we propose a novel end-to-end deep graph representation learning (Deep Multimodal Brain Networks - DMBN) to fuse multimodal brain networks. Specifically, we decipher the cross-modality relationship through a graph encoding and decoding process. The higher-order network mappings from brain structural networks to functional networks are learned in the node domain. The learned network representation is a set of node features that are informative to induce brain saliency maps in a supervised manner. We test our framework in both synthetic and real image data. The experimental results show the superiority of the proposed method over some other state-of-the-art deep brain network models.
Exploiting Partial Common Information Microstructure for Multi-Modal Brain Tumor Segmentation

Yongsheng Mei,Guru Venkataramani,Tian Lan

2023-07-15

Abstract:Learning with multiple modalities is crucial for automated brain tumor segmentation from magnetic resonance imaging data. Explicitly optimizing the common information shared among all modalities (e.g., by maximizing the total correlation) has been shown to achieve better feature representations and thus enhance the segmentation performance. However, existing approaches are oblivious to partial common information shared by subsets of the modalities. In this paper, we show that identifying such partial common information can significantly boost the discriminative power of image segmentation models. In particular, we introduce a novel concept of partial common information mask (PCI-mask) to provide a fine-grained characterization of what partial common information is shared by which subsets of the modalities. By solving a masked correlation maximization and simultaneously learning an optimal PCI-mask, we identify the latent microstructure of partial common information and leverage it in a self-attention module to selectively weight different feature representations in multi-modal data. We implement our proposed framework on the standard U-Net. Our experimental results on the Multi-modal Brain Tumor Segmentation Challenge (BraTS) datasets outperform those of state-of-the-art segmentation baselines, with validation Dice similarity coefficients of 0.920, 0.897, 0.837 for the whole tumor, tumor core, and enhancing tumor on BraTS-2020.

Computer Vision and Pattern Recognition,Machine Learning
Multimodal 3D Image Registration for Mapping Brain Disorders

Hassan Mahmood,Syed Mohammed Shamsul Islam,Asim Iqbal

DOI: https://doi.org/10.1101/2024.08.24.609508

2024-08-26

Abstract:We introduce an AI-driven approach for robust 3D brain image registration, addressing challenges posed by diverse hardware scanners and imaging sites. Our model trained using an SSIM-driven loss function, prioritizes structural coherence over voxel-wise intensity matching, making it uniquely robust to inter-scanner and intra-modality variations. This innovative end-to-end framework combines global alignment and non-rigid registration modules, specifically designed to handle structural, intensity, and domain variances in 3D brain imaging data. Our approach outperforms the baseline model in handling these shifts, achieving results that align closely with clinical ground-truth measurements. We demonstrate its efficacy on 3D brain data from healthy individuals and dementia patients, with particular success in quantifying brain atrophy, a key biomarker for Alzheimer's disease and other brain disorders. By effectively managing variability in multi-site, multi-scanner neuroimaging studies, our approach enhances the precision of atrophy measurements for clinical trials and longitudinal studies. This advancement promises to improve diagnostic and prognostic capabilities for neurodegenerative disorders.

Neuroscience
Anatomically Constrained and Attention-Guided Deep Feature Fusion for Joint Segmentation and Deformable Medical Image Registration.

Hee Guan Khor,Guochen Ning,Yihua Sun,Xu Lu,Xinran Zhang,Hongen Liao

DOI: https://doi.org/10.1016/j.media.2023.102811

IF: 10.9

2023-01-01

Medical Image Analysis

Abstract:The main objective of anatomically plausible results for deformable image registration is to improve model's registration accuracy by minimizing the difference between a pair of fixed and moving images. Since many anatomical features are closely related to each other, leveraging supervision from auxiliary tasks (such as supervised anatomical segmentation) has the potential to enhance the realism of the warped images after registration. In this work, we employ a Multi-Task Learning framework to formulate registration and segmentation as a joint issue, in which we utilize anatomical constraint from auxiliary supervised segmentation to enhance the realism of the predicted images. First, we propose a Cross-Task Attention Block to fuse the high-level feature from both the registration and segmentation network. With the help of initial anatomical segmentation, the registration network can benefit from learning the task-shared feature correlation and rapidly focusing on the parts that need deformation. On the other hand, the anatomical segmentation discrepancy from ground-truth fixed annotations and predicted segmentation maps of initial warped images are integrated into the loss function to guide the convergence of the registration network. Ideally, a good deformation field should be able to minimize the loss function of registration and segmentation. The voxel-wise anatomical constraint inferred from segmentation helps the registration network to reach a global optimum for both deformable and segmentation learning. Both networks can be employed independently during the testing phase, enabling only the registration output to be predicted when the segmentation labels are unavailable. Qualitative and quantitative results indicate that our proposed methodology significantly outperforms the previous state-of-the-art approaches on inter-patient brain MRI registration and pre- and intra-operative uterus MRI registration tasks within our specific experimental setup, which leads to state-of-the-art registration quality scores of 0.755 and 0.731 (i.e., by 0.8% and 0.5% increases) DSC for both tasks, respectively.
Deep Cascaded Attention Network for Multi-task Brain Tumor Segmentation.

Hai Xu,Hongtao Xie,Yizhi Liu,Chuandong Cheng,Chaoshi Niu,Yongdong Zhang

DOI: https://doi.org/10.1007/978-3-030-32248-9_47

2019-01-01

Abstract:Multi-modal magnetic resonance images (MRIs) play an important role in the diagnosis and treatment of brain tumors. Due to heterogeneous diversities, it's of great challenge to segment gliomas into hierarchical regions. Decomposing the multi-class segmentation task into sequential subtasks with cascaded models has proved its effectiveness, but leads to model redundancy, training complexity, and task isolation. In this paper, we propose a simple yet efficient 3D deep cascaded attention network (DCAN) for brain tumor segmentation. Specifically, we settle multi-tasks into corresponding branches with a shared feature extractor to reduce model complexity. Second, instead of explicitly extracting spatial evolutionary relationships of sub-regions using several consecutive models, a cascaded attention mechanism is introduced to implicitly involve potential subregions correlations as guidance. Moreover, we present a feature bridge module (FBM) to narrow feature fusion gaps. Thus, DCAN is able to capture the hierarchical correlations of overlapping regions and simultaneously tackle multi-tasks in a single model. The comprehensive experimental comparisons on the BRATS 2018 dataset show DCAN achieves top performance with dice scores of 81.71%, 91.18% and 86.19% for the enhancing tumor, whole tumor and tumor core, respectively.
Modality-Pairing Learning for Brain Tumor Segmentation

Yixin Wang,Yao Zhang,Feng Hou,Yang Liu,Jiang Tian,Cheng Zhong,Yang Zhang,Zhiqiang He

DOI: https://doi.org/10.48550/arXiv.2010.09277

2020-12-29

Abstract:Automatic brain tumor segmentation from multi-modality Magnetic Resonance Images (MRI) using deep learning methods plays an important role in assisting the diagnosis and treatment of brain tumor. However, previous methods mostly ignore the latent relationship among different modalities. In this work, we propose a novel end-to-end Modality-Pairing learning method for brain tumor segmentation. Paralleled branches are designed to exploit different modality features and a series of layer connections are utilized to capture complex relationships and abundant information among modalities. We also use a consistency loss to minimize the prediction variance between two branches. Besides, learning rate warmup strategy is adopted to solve the problem of the training instability and early over-fitting. Lastly, we use average ensemble of multiple models and some post-processing techniques to get final results. Our method is tested on the BraTS 2020 online testing dataset, obtaining promising segmentation performance, with average dice scores of 0.891, 0.842, 0.816 for the whole tumor, tumor core and enhancing tumor, respectively. We won the second place of the BraTS 2020 Challenge for the tumor segmentation task.

Image and Video Processing,Computer Vision and Pattern Recognition
Brain Tumor Segmentation via Multi-Modalities Interactive Feature Learning

Bo Wang,Jingyi Yang,Hong Peng,Jingyang Ai,Lihua An,Bo Yang,Zheng You,Lin Ma

DOI: https://doi.org/10.3389/fmed.2021.653925

IF: 3.9

2021-05-13

Frontiers in Medicine

Abstract:Automatic segmentation of brain tumors from multi-modalities magnetic resonance image data has the potential to enable preoperative planning and intraoperative volume measurement. Recent advances in deep convolutional neural network technology have opened up an opportunity to achieve end-to-end segmenting the brain tumor areas. However, the medical image data used in brain tumor segmentation are relatively scarce and the appearance of brain tumors is varied, so that it is difficult to find a learnable pattern to directly describe tumor regions. In this paper, we propose a novel cross-modalities interactive feature learning framework to segment brain tumors from the multi-modalities data. The core idea is that the multi-modality MR data contain rich patterns of the normal brain regions, which can be easily captured and can be potentially used to detect the non-normal brain regions, i.e., brain tumor regions. The proposed multi-modalities interactive feature learning framework consists of two modules: cross-modality feature extracting module and attention guided feature fusing module, which aim at exploring the rich patterns cross multi-modalities and guiding the interacting and the fusing process for the rich features from different modalities. Comprehensive experiments are conducted on the BraTS 2018 benchmark, which show that the proposed cross-modality feature learning framework can effectively improve the brain tumor segmentation performance when compared with the baseline methods and state-of-the-art methods.

medicine, general & internal

On a simple definition of computable function of a real variable-with applications to functions of a complex variable

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Accurate Whole-Brain Segmentation for Bimodal PET/MR Images Via a Cross-Attention Mechanism

Deep learning-based 3D brain multimodal medical image registration

Cross-Modality Coherent Registration of Whole Mouse Brains

A hybrid deep learning framework for integrated segmentation and registration: evaluation on longitudinal white matter tract changes

Cross-modal coherent registration of whole mouse brains

Unsupervised deep learning registration model for multimodal brain images

Deep learning facilitates fully automated brain image registration of optoacoustic tomography and magnetic resonance imaging

A novel brain registration model combining structural and functional MRI information

Multimodal Brain Tumor Segmentation Boosted by Monomodal Normal Brain Images

Metabolic fate of 6,6,9-trimethyl-9-azabicyclo(3,3,1)non-3 -yl , -di(2-thienyl)glycolate hydrochloride monohydrate (PG-501).

A Context-Supported Deep Learning Framework for Multimodal Brain Imaging Classification

The Method of Multimodal MRI Brain Image Segmentation Based on Differential Geometric Features

Deep Representation Learning For Multimodal Brain Networks

Exploiting Partial Common Information Microstructure for Multi-Modal Brain Tumor Segmentation

Multimodal 3D Image Registration for Mapping Brain Disorders

Anatomically Constrained and Attention-Guided Deep Feature Fusion for Joint Segmentation and Deformable Medical Image Registration.

Deep Cascaded Attention Network for Multi-task Brain Tumor Segmentation.

Modality-Pairing Learning for Brain Tumor Segmentation

Brain Tumor Segmentation via Multi-Modalities Interactive Feature Learning