Abstract:ABSTRACT Brain tumor segmentation is paramount in medical diagnostics. This study presents a multistage segmentation model consisting of two main steps. First, the fusion of magnetic resonance imaging (MRI) modalities creates new and more effective tumor imaging modalities. Second, the semantic segmentation of the original and fused modalities, utilizing various modified architectures of the U‐Net model. In the first step, a residual network with multi‐scale backbone architecture (Res2Net) and guided filter are employed for pixel‐by‐pixel image fusion tasks without requiring any training or learning process. This method captures both detailed and base elements from the multimodal images to produce better and more informative fused images that significantly enhance the segmentation process. Many fusion scenarios were performed and analyzed, revealing that the best fusion results are attained when combining T2‐weighted (T2) with fluid‐attenuated inversion recovery (FLAIR) and T1‐weighted contrast‐enhanced (T1CE) with FLAIR modalities. In the second step, several models, including the U‐Net and its many modifications (adding attention layers, residual connections, and depthwise separable connections), are trained using both the original and fused modalities. Further, a “Model Selection‐based” fusion of these individual models is also considered for more enhancement. In the preprocessing step, the images are resized by cropping them to decrease the pixel count and minimize background interference. Experiments utilizing the brain tumor segmentation (BraTS) 2020 dataset were performed to verify the efficiency and accuracy of the proposed methodology. The “Model Selection‐based” fusion model achieved an average Dice score of 88.4%, an individual score of 91.1% for the whole tumor (WT) class, an average sensitivity score of 86.26%, and a specificity score of 91.7%. These results prove the robustness and high performance of the proposed methodology compared to other state‐of‐the‐art methods.

Multimodal Medical Imaging Using Modern Deep Learning Approaches

Deep Learning-Based Image Segmentation on Multimodal Medical Imaging

Multimodal medical image fusion and classification using deep learning techniques

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

Multi-modal Medical Data Fusion using Deep Learning

Multimodal medical image fusion algorithm in the era of big data

Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

Multimodal Medical Image Fusion: The Perspective of Deep Learning

Multi Modalities Medical Image Fusion Using Deep Learning and Metaverse Technology: Healthcare 4.0 A Futuristic Approach

Deep Multimodal Guidance for Medical Image Classification

Multimodal Fusion Learning with Dual Attention for Medical Imaging

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Multi-modal fusion deep transfer learning for accurate brain tumor classification using magnetic resonance imaging images

Multimodal Medical Image Fusion Techniques – A Review

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound images

Multimodal Fusion for Enhanced Semantic Segmentation in Brain Tumor Imaging: Integrating Deep Learning and Guided Filtering Via Advanced 3D Semantic Segmentation Architectures

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

A review: Deep learning for medical image segmentation using multi-modality fusion

MM-BiFPN: Multi-Modality Fusion Network With Bi-FPN for MRI Brain Tumor Segmentation

An Attentive Multi-Modal CNN for Brain Tumor Radiogenomic Classification