Abstract:In the domain of multi-focus (MF) and multi-model image fusion (MMIF), accurately merging focused regions from various images remains a challenge. Existing methods often fall short of capturing the nuances of depth-of-field variations, leading to sub-optimal fusion outcomes. This research introduces a novel approach, utilizing both supervised and unsupervised deep learning techniques. Unlike conventional methods, the proposed model explores the realm of deep learning. to decode sharpness dynamics and activity levels, thereby enhancing the fusion process. This innovative fusion method aims to bridge the gap between modern and traditional image processing techniques, promising a breakthrough in MF image fusion and MMIF performance. Previous methods have struggled with depth-of-field variations and failed to achieve optimal fusion outcomes in the context of multi-focus and multi-model image fusion (MF and MMIF). The existing approaches cannot often capture the intricate details required for accurate merging, highlighting the need for a more advanced solution. The proposed solution introduces a novel semi-supervised deep learning model utilizing spatial frequency analysis for brain image fusion. A convolutional neural network (CNN) is employed for feature extraction, specifically designed for learning MF and multi-model images. Spatial frequency is leveraged to assess the activity level of deep features, ensuring enhanced fusion outcomes. The CNN architecture is optimized with a 5-convolutional layer cascade, omitting pooling layers for efficient deep feature extraction. The fusion strategy incorporates decision maps derived from spatial frequency-based consistency validation, contributing to improved accuracy. Evaluation using SSIM and PSNR against various fusion methods demonstrates the superior performance of the proposed model. Experimental results show that the proposed model outperforms existing methods, demonstrating its effectiveness in brain image fusion.

Multimodal Contrastive Learning for Brain-Machine Fusion: from Brain-in-the-loop Modeling to Brain-out-of-the-loop Application

Brain-inspired Multimodal Learning Based on Neural Networks

Multimodal Fusion of Brain Imaging Data: Methods and Applications

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

NeuralOOD: Improving Out-of-Distribution Generalization Performance with Brain-machine Fusion Learning Framework

Multimodal foundation models are better simulators of the human brain

A Context-Supported Deep Learning Framework for Multimodal Brain Imaging Classification

Interpretable Multimodal Fusion Networks Reveal Mechanisms of Brain Cognition

Improved Classification Performance of EEG-fNIRS Multimodal Brain-Computer Interface Based on Multi-Domain Features and Multi-Level Progressive Learning

MCFHNet: Multi-Channel Fusion Hybrid Network for Efficient EEG-fNIRS Multi-modal Motor Imagery Decoding

Multiscale Spatial-Temporal Feature Fusion Neural Network for Motor Imagery Brain-Computer Interfaces

SpFusionNet: deep learning-driven brain image fusion with spatial frequency analysis

On the Benefits of Early Fusion in Multimodal Representation Learning

A novel multimodal approach for hybrid brain-computer interface

Early-stage fusion of EEG and fNIRS improves classification of motor imagery

Bidirectional Mapping with Contrastive Learning on Multimodal Neuroimaging Data

Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment

MSTGC: Multi-Channel Spatio-Temporal Graph Convolution Network for Multi-Modal Brain Networks Fusion

Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI

MBCFNet: A Multimodal Brain–Computer Fusion Network for Human Intention Recognition