Photoperiodic regulation of the diapause of the progeny in Trichogramma embryophagum Htg. (Hymenoptera, Trichogrammatidae): Dynamics of sensitivity to photoperiod at the immature stages of maternal females

M. F. Ivanov,S. Reznik

DOI: https://doi.org/10.1134/S0013873808030019

2008-06-24

Entomological Review

Abstract:

What problem does this paper attempt to address?

AMLP:Adaptive Masking Lesion Patches for Self-supervised Medical Image Segmentation

Xiangtao Wang,Ruizhi Wang,Jie Zhou,Thomas Lukasiewicz,Zhenghua Xu

DOI: https://doi.org/10.48550/arXiv.2309.04312

2023-09-08

Abstract:Self-supervised masked image modeling has shown promising results on natural images. However, directly applying such methods to medical images remains challenging. This difficulty stems from the complexity and distinct characteristics of lesions compared to natural images, which impedes effective representation learning. Additionally, conventional high fixed masking ratios restrict reconstructing fine lesion details, limiting the scope of learnable information. To tackle these limitations, we propose a novel self-supervised medical image segmentation framework, Adaptive Masking Lesion Patches (AMLP). Specifically, we design a Masked Patch Selection (MPS) strategy to identify and focus learning on patches containing lesions. Lesion regions are scarce yet critical, making their precise reconstruction vital. To reduce misclassification of lesion and background patches caused by unsupervised clustering in MPS, we introduce an Attention Reconstruction Loss (ARL) to focus on hard-to-reconstruct patches likely depicting lesions. We further propose a Category Consistency Loss (CCL) to refine patch categorization based on reconstruction difficulty, strengthening distinction between lesions and background. Moreover, we develop an Adaptive Masking Ratio (AMR) strategy that gradually increases the masking ratio to expand reconstructible information and improve learning. Extensive experiments on two medical segmentation datasets demonstrate AMLP's superior performance compared to existing self-supervised approaches. The proposed strategies effectively address limitations in applying masked modeling to medical images, tailored to capturing fine lesion details vital for segmentation tasks.

Computer Vision and Pattern Recognition
MPS-AMS: Masked Patches Selection and Adaptive Masking Strategy Based Self-Supervised Medical Image Segmentation

Xiangtao Wang,Ruizhi Wang,Biao Tian,Jiaojiao Zhang,Shuo Zhang,Junyang Chen,Thomas Lukasiewicz,Zhenghua Xu

DOI: https://doi.org/10.48550/arXiv.2302.13699

2023-02-27

Computer Vision and Pattern Recognition

Abstract:Existing self-supervised learning methods based on contrastive learning and masked image modeling have demonstrated impressive performances. However, current masked image modeling methods are mainly utilized in natural images, and their applications in medical images are relatively lacking. Besides, their fixed high masking strategy limits the upper bound of conditional mutual information, and the gradient noise is considerable, making less the learned representation information. Motivated by these limitations, in this paper, we propose masked patches selection and adaptive masking strategy based self-supervised medical image segmentation method, named MPS-AMS. We leverage the masked patches selection strategy to choose masked patches with lesions to obtain more lesion representation information, and the adaptive masking strategy is utilized to help learn more mutual information and improve performance further. Extensive experiments on three public medical image segmentation datasets (BUSI, Hecktor, and Brats2018) show that our proposed method greatly outperforms the state-of-the-art self-supervised baselines.
AnatoMask: Enhancing Medical Image Segmentation with Reconstruction-guided Self-masking

Yuheng Li,Tianyu Luan,Yizhou Wu,Shaoyan Pan,Yenho Chen,Xiaofeng Yang

2024-07-17

Abstract:Due to the scarcity of labeled data, self-supervised learning (SSL) has gained much attention in 3D medical image segmentation, by extracting semantic representations from unlabeled data. Among SSL strategies, Masked image modeling (MIM) has shown effectiveness by reconstructing randomly masked images to learn detailed representations. However, conventional MIM methods require extensive training data to achieve good performance, which still poses a challenge for medical imaging. Since random masking uniformly samples all regions within medical images, it may overlook crucial anatomical regions and thus degrade the pretraining efficiency. We propose AnatoMask, a novel MIM method that leverages reconstruction loss to dynamically identify and mask out anatomically significant regions to improve pretraining efficacy. AnatoMask takes a self-distillation approach, where the model learns both how to find more significant regions to mask and how to reconstruct these masked regions. To avoid suboptimal learning, Anatomask adjusts the pretraining difficulty progressively using a masking dynamics function. We have evaluated our method on 4 public datasets with multiple imaging modalities (CT, MRI, and PET). AnatoMask demonstrates superior performance and scalability compared to existing SSL methods. The code is available at <a class="link-external link-https" href="https://github.com/ricklisz/AnatoMask" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
When Masked Image Modeling Meets Source-free Unsupervised Domain Adaptation: Dual-Level Masked Network for Semantic Segmentation

Gang Li,Xianzheng Ma,Zhao Wang,Hao Li,Qifei Zhang,Chao Wu

DOI: https://doi.org/10.1145/3581783.3612521

2023-01-01

Abstract:Source-Free domain adaptive Semantic Segmentation (SFSS) aims to transfer knowledge from source domain to the target domain with only pre-trained source segmentation model and the unlabeled target dataset. Only a few works have been researched for SFSS, relying on entropy minimization, pseudo-labeling. Nevertheless, due to the domain bias, these methods tend to suffering from the confusion of classes with a similar visual appearance in different domains. To address the above issue, we propose to enhance discriminability towards target samples with masked image modeling to model spatial context relations as additional recognition clues. Specifically, we design a novel Dual-Level Masked Consistency method, which explicitly encourages the model to learn comprehensive context relations, i.e. patch-wise context and channel-wise context, on the target domain. By randomly masking target images and forcing the model to reconstruct predictions of the entire image with left unmasked part, the model has to make full use of spatially contextual information. To take a step further, we propose a novel masking strategy considering both local context and global context information by applying patch-wise masking on image patches and channel-wise masking on latent features. Notably, patch-wise context learning and channel-wise context learning can complement each other. Extensive experiments demonstrate the effectiveness of our proposed method and our method achieves state-of-the-art performance on two synthetic-to-real benchmarks: GTA5→Cityscapes and SYNTHIA→Cityscapes.
HybridMIM: A Hybrid Masked Image Modeling Framework for 3D Medical Image Segmentation

Zhaohu Xing,Lei Zhu,Lequan Yu,Zhiheng Xing,Liang Wan

DOI: https://doi.org/10.1109/JBHI.2024.3360239

2023-03-18

Abstract:Masked image modeling (MIM) with transformer backbones has recently been exploited as a powerful self-supervised pre-training technique. The existing MIM methods adopt the strategy to mask random patches of the image and reconstruct the missing pixels, which only considers semantic information at a lower level, and causes a long pre-training time.This paper presents HybridMIM, a novel hybrid self-supervised learning method based on masked image modeling for 3D medical image segmentation.Specifically, we design a two-level masking hierarchy to specify which and how patches in sub-volumes are masked, effectively providing the constraints of higher level semantic information. Then we learn the semantic information of medical images at three levels, including:1) partial region prediction to reconstruct key contents of the 3D image, which largely reduces the pre-training time burden (pixel-level); 2) patch-masking perception to learn the spatial relationship between the patches in each sub-volume (region-level).and 3) drop-out-based contrastive learning between samples within a mini-batch, which further improves the generalization ability of the framework (sample-level). The proposed framework is versatile to support both CNN and transformer as encoder backbones, and also enables to pre-train decoders for image segmentation. We conduct comprehensive experiments on four widely-used public medical image segmentation datasets, including BraTS2020, BTCV, MSD Liver, and MSD Spleen. The experimental results show the clear superiority of HybridMIM against competing supervised methods, masked pre-training approaches, and other self-supervised methods, in terms of quantitative metrics, timing performance and qualitative observations. The codes of HybridMIM are available at <a class="link-external link-https" href="https://github.com/ge-xing/HybridMIM" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition
Masked Image Modeling Advances 3D Medical Image Analysis

Zekai Chen,Devansh Agarwal,Kshitij Aggarwal,Wiem Safta,Samit Hirawat,Venkat Sethuraman,Mariann Micsinai Balan,Kevin Brown

DOI: https://doi.org/10.48550/arXiv.2204.11716

2022-08-24

Abstract:Recently, masked image modeling (MIM) has gained considerable attention due to its capacity to learn from vast amounts of unlabeled data and has been demonstrated to be effective on a wide variety of vision tasks involving natural images. Meanwhile, the potential of self-supervised learning in modeling 3D medical images is anticipated to be immense due to the high quantities of unlabeled images, and the expense and difficulty of quality labels. However, MIM's applicability to medical images remains uncertain. In this paper, we demonstrate that masked image modeling approaches can also advance 3D medical images analysis in addition to natural images. We study how masked image modeling strategies leverage performance from the viewpoints of 3D medical image segmentation as a representative downstream task: i) when compared to naive contrastive learning, masked image modeling approaches accelerate the convergence of supervised training even faster (1.40$\times$) and ultimately produce a higher dice score; ii) predicting raw voxel values with a high masking ratio and a relatively smaller patch size is non-trivial self-supervised pretext-task for medical images modeling; iii) a lightweight decoder or projection head design for reconstruction is powerful for masked image modeling on 3D medical images which speeds up training and reduce cost; iv) finally, we also investigate the effectiveness of MIM methods under different practical scenarios where different image resolutions and labeled data ratios are applied.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning,Other Quantitative Biology
GMIM: Self-supervised pre-training for 3D medical image segmentation with adaptive and hierarchical masked image modeling

Liangce Qi,Zhengang Jiang,Weili Shi,Feng Qu,Guanyuan Feng

DOI: https://doi.org/10.1016/j.compbiomed.2024.108547

Abstract:Self-supervised pre-training and fully supervised fine-tuning paradigms have received much attention to solve the data annotation problem in deep learning fields. Compared with traditional pre-training on large natural image datasets, medical self-supervised learning methods learn rich representations derived from unlabeled data itself thus avoiding the distribution shift between different image domains. However, nowadays state-of-the-art medical pre-training methods were specifically designed for downstream tasks making them less flexible and difficult to apply to new tasks. In this paper, we propose grid mask image modeling, a flexible and general self-supervised method to pre-train medical vision transformers for 3D medical image segmentation. Our goal is to guide networks to learn the correlations between organs and tissues by reconstructing original images based on partial observations. The relationships are consistent within the human body and invariant to disease type or imaging modality. To achieve this, we design a Siamese framework consisting of an online branch and a target branch. An adaptive and hierarchical masking strategy is employed in the online branch to (1) learn the boundaries or small contextual mutation regions within images; (2) to learn high-level semantic representations from deeper layers of the multiscale encoder. In addition, the target branch provides representations for contrastive learning to further reduce representation redundancy. We evaluate our method through segmentation performance on two public datasets. The experimental results demonstrate our method outperforms other self-supervised methods. Codes are available at https://github.com/mobiletomb/Gmim.
Rethinking masked image modelling for medical image representation

Yutong Xie,Lin Gu,Tatsuya Harada,Jianpeng Zhang,Yong Xia,Qi Wu

DOI: https://doi.org/10.1016/j.media.2024.103304

2024-08-17

Abstract:Masked Image Modelling (MIM), a form of self-supervised learning, has garnered significant success in computer vision by improving image representations using unannotated data. Traditional MIMs typically employ a strategy of random sampling across the image. However, this random masking technique may not be ideally suited for medical imaging, which possesses distinct characteristics divergent from natural images. In medical imaging, particularly in pathology, disease-related features are often exceedingly sparse and localized, while the remaining regions appear normal and undifferentiated. Additionally, medical images frequently accompany reports, directly pinpointing pathological changes' location. Inspired by this, we propose Masked medical Image Modelling (MedIM), a novel approach, to our knowledge, the first research that employs radiological reports to guide the masking and restore the informative areas of images, encouraging the network to explore the stronger semantic representations from medical images. We introduce two mutual comprehensive masking strategies, knowledge-driven masking (KDM), and sentence-driven masking (SDM). KDM uses Medical Subject Headings (MeSH) words unique to radiology reports to identify symptom clues mapped to MeSH words (e.g., cardiac, edema, vascular, pulmonary) and guide the mask generation. Recognizing that radiological reports often comprise several sentences detailing varied findings, SDM integrates sentence-level information to identify key regions for masking. MedIM reconstructs images informed by this masking from the KDM and SDM modules, promoting a comprehensive and enriched medical image representation. Our extensive experiments on seven downstream tasks covering multi-label/class image classification, pneumothorax segmentation, and medical image-report analysis, demonstrate that MedIM with report-guided masking achieves competitive performance. Our method substantially outperforms ImageNet pre-training, MIM-based pre-training, and medical image-report pre-training counterparts. Codes are available at https://github.com/YtongXie/MedIM.
FreMIM: Fourier Transform Meets Masked Image Modeling for Medical Image Segmentation

Wenxuan Wang,Jing Wang,Chen Chen,Jianbo Jiao,Yuanxiu Cai,Shanshan Song,Jiangyun Li

2023-11-30

Abstract:The research community has witnessed the powerful potential of self-supervised Masked Image Modeling (MIM), which enables the models capable of learning visual representation from unlabeled data. In this paper, to incorporate both the crucial global structural information and local details for dense prediction tasks, we alter the perspective to the frequency domain and present a new MIM-based framework named FreMIM for self-supervised pre-training to better accomplish medical image segmentation tasks. Based on the observations that the detailed structural information mainly lies in the high-frequency components and the high-level semantics are abundant in the low-frequency counterparts, we further incorporate multi-stage supervision to guide the representation learning during the pre-training phase. Extensive experiments on three benchmark datasets show the superior advantage of our FreMIM over previous state-of-the-art MIM methods. Compared with various baselines trained from scratch, our FreMIM could consistently bring considerable improvements to model performance. The code will be publicly available at <a class="link-external link-https" href="https://github.com/Rubics-Xuan/FreMIM" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation

Yangyang Li,Xuanting Hao,Ronghua Shang,Licheng Jiao

2024-11-14

Abstract:In view of the fact that semi- and self-supervised learning share a fundamental principle, effectively modeling knowledge from unlabeled data, various semi-supervised semantic segmentation methods have integrated representative self-supervised learning paradigms for further regularization. However, the potential of the state-of-the-art generative self-supervised paradigm, masked image modeling, has been scarcely studied. This paradigm learns the knowledge through establishing connections between the masked and visible parts of masked image, during the pixel reconstruction process. By inheriting and extending this insight, we successfully leverage masked image modeling to boost semi-supervised semantic segmentation. Specifically, we introduce a novel class-wise masked image modeling that independently reconstructs different image regions according to their respective classes. In this way, the mask-induced connections are established within each class, mitigating the semantic confusion that arises from plainly reconstructing images in basic masked image modeling. To strengthen these intra-class connections, we further develop a feature aggregation strategy that minimizes the distances between features corresponding to the masked and visible parts within the same class. Additionally, in semantic space, we explore the application of masked image modeling to enhance regularization. Extensive experiments conducted on well-known benchmarks demonstrate that our approach achieves state-of-the-art performance. The code will be available at <a class="link-external link-https" href="https://github.com/haoxt/S4MIM" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
FedATA: Adaptive Attention Aggregation for Federated Self-supervised Medical Image Segmentation

Jian Dai,Hao Wu,Huan Liu,Liheng Yu,Xing Hu,Xiao Liu,Daoying Geng

DOI: https://doi.org/10.1016/j.neucom.2024.128691

IF: 6

2025-01-01

Neurocomputing

Abstract:Pre-trained on large-scale datasets has profoundly promoted the development of deep learning models in medical image analysis. For medical image segmentation, collecting a large number of labeled volumetric medical images from multiple institutions is an enormous challenge due to privacy concerns. Self-supervised learning with mask image modeling (MIM) can learn general representation without annotations. Integrating MIM into FL enables collaborative learning of an efficient pre-trained model from unlabeled data, followed by fine-tuning with limited annotations. However, setting pixels as reconstruction targets in traditional MIM fails to facilitate robust representation learning due to the medical image's complexity and distinct characteristics. On the other hand, the generalization of the aggregated model in FL is also impaired under the heterogeneous data distributions among institutions. To address these issues, we proposed a novel self-supervised federated learning, which combines masked self-distillation with adaptive attention federated learning. Such incorporation enjoys two vital benefits. First, masked self-distillation sets high-quality latent representations of masked tokens as the target, improving the descriptive capability of the learned presentation rather than reconstructing low-level pixels. Second, adaptive attention aggregation with Personalized federate learning effectively captures specific-related representation from the aggregated model, thus facilitating local fine-tuning performance for target tasks. We conducted comprehensive experiments on two medical segmentation tasks using a large-scale dataset consisting of volumetric medical images from multiple institutions, demonstrating superior performance compared to existing federated self-supervised learning approaches.
Self-Supervised Masking for Unsupervised Anomaly Detection and Localization

Chaoqin Huang,Qinwei Xu,Yanfeng Wang,Yu Wang,Ya Zhang

DOI: https://doi.org/10.48550/arXiv.2205.06568

2022-05-13

Computer Vision and Pattern Recognition

Abstract:Recently, anomaly detection and localization in multimedia data have received significant attention among the machine learning community. In real-world applications such as medical diagnosis and industrial defect detection, anomalies only present in a fraction of the images. To extend the reconstruction-based anomaly detection architecture to the localized anomalies, we propose a self-supervised learning approach through random masking and then restoring, named Self-Supervised Masking (SSM) for unsupervised anomaly detection and localization. SSM not only enhances the training of the inpainting network but also leads to great improvement in the efficiency of mask prediction at inference. Through random masking, each image is augmented into a diverse set of training triplets, thus enabling the autoencoder to learn to reconstruct with masks of various sizes and shapes during training. To improve the efficiency and effectiveness of anomaly detection and localization at inference, we propose a novel progressive mask refinement approach that progressively uncovers the normal regions and finally locates the anomalous regions. The proposed SSM method outperforms several state-of-the-arts for both anomaly detection and anomaly localization, achieving 98.3% AUC on Retinal-OCT and 93.9% AUC on MVTec AD, respectively.
MedIM: Boost Medical Image Representation via Radiology Report-Guided Masking

Yutong Xie,Lin Gu,Tatsuya Harada,Jianpeng Zhang,Yong Xia,Qi Wu

DOI: https://doi.org/10.1007/978-3-031-43907-0_2

2023-01-01

Abstract:Masked image modelling (MIM)-based pre-training shows promise in improving image representations with limited annotated data by randomly masking image patches and reconstructing them. However, random masking may not be suitable for medical images due to their unique pathology characteristics. This paper proposes Masked medical Image Modelling (MedIM), a novel approach, to our knowledge, the first research that masks and reconstructs discriminative areas guided by radiological reports, encouraging the network to explore the stronger semantic representations from medical images. We introduce two mutual comprehensive masking strategies, knowledge word-driven masking (KWM) and sentence-driven masking (SDM). KWM uses Medical Subject Headings (MeSH) words unique to radiology reports to identify discriminative cues mapped to MeSH words and guide the mask generation. SDM considers that reports usually have multiple sentences, each of which describes different findings, and therefore integrates sentencelevel information to identify discriminative regions for mask generation. MedIM integrates both strategies by simultaneously restoring the images masked by KWM and SDM for a more robust and representative medical visual representation. Our extensive experiments on various downstream tasks covering multi-label/class image classification, medical image segmentation, and medical image-text analysis, demonstrate that MedIM with report-guided masking achieves competitive performance. Our method substantially outperforms ImageNet pre-training, MIM-based pre-training, and medical image-report pre-training counterparts. Codes are available at https://github.com/YtongXie/MedIM.
Medical supervised masked autoencoders: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification

Jiawei Mao,Shujian Guo,Yuanqi Chang,Xuesong Yin,Binling Nie

DOI: https://doi.org/10.48550/arXiv.2305.05871

2023-05-10

Abstract:Masked autoencoders (MAEs) have displayed significant potential in the classification and semantic segmentation of medical images in the last year. Due to the high similarity of human tissues, even slight changes in medical images may represent diseased tissues, necessitating fine-grained inspection to pinpoint diseased tissues. The random masking strategy of MAEs is likely to result in areas of lesions being overlooked by the model. At the same time, inconsistencies between the pre-training and fine-tuning phases impede the performance and efficiency of MAE in medical image classification. To address these issues, we propose a medical supervised masked autoencoder (MSMAE) in this paper. In the pre-training phase, MSMAE precisely masks medical images via the attention maps obtained from supervised training, contributing to the representation learning of human tissue in the lesion area. During the fine-tuning phase, MSMAE is also driven by attention to the accurate masking of medical images. This improves the computational efficiency of the MSMAE while increasing the difficulty of fine-tuning, which indirectly improves the quality of MSMAE medical diagnosis. Extensive experiments demonstrate that MSMAE achieves state-of-the-art performance in case with three official medical datasets for various diseases. Meanwhile, transfer learning for MSMAE also demonstrates the great potential of our approach for medical semantic segmentation tasks. Moreover, the MSMAE accelerates the inference time in the fine-tuning phase by 11.2% and reduces the number of floating-point operations (FLOPs) by 74.08% compared to a traditional MAE.

Computer Vision and Pattern Recognition
Advancing Volumetric Medical Image Segmentation via Global-Local Masked Autoencoder

Jia-Xin Zhuang,Luyang Luo,Hao Chen

2023-08-24

Abstract:Masked autoencoder (MAE) is a promising self-supervised pre-training technique that can improve the representation learning of a neural network without human intervention. However, applying MAE directly to volumetric medical images poses two challenges: (i) a lack of global information that is crucial for understanding the clinical context of the holistic data, (ii) no guarantee of stabilizing the representations learned from randomly masked inputs. To address these limitations, we propose the \textbf{G}lobal-\textbf{L}ocal \textbf{M}asked \textbf{A}uto\textbf{E}ncoder (GL-MAE), a simple yet effective self-supervised pre-training strategy. In addition to reconstructing masked local views, as in previous methods, GL-MAE incorporates global context learning by reconstructing masked global views. Furthermore, a complete global view is integrated as an anchor to guide the reconstruction and stabilize the learning process through global-to-global consistency learning and global-to-local consistency learning. Finetuning results on multiple datasets demonstrate the superiority of our method over other state-of-the-art self-supervised algorithms, highlighting its effectiveness on versatile volumetric medical image segmentation tasks, even when annotations are scarce. Our codes and models will be released upon acceptance.

Computer Vision and Pattern Recognition,Artificial Intelligence
Evolved Hierarchical Masking for Self-Supervised Learning

Zhanzhou Feng,Shiliang Zhang

DOI: https://doi.org/10.1109/tpami.2024.3490776

IF: 23.6

2024-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Existing Masked Image Modeling methods apply fixed mask patterns to guide the self-supervised training. As those mask patterns resort to different criteria to depict image contents, sticking to a fixed pattern leads to a limited vision cues modeling capability. This paper introduces an evolved hierarchical masking method to pursue general visual cues modeling in self-supervised learning. The proposed method leverages the vision model being trained to parse the input visual cues into a hierarchy structure, which is hence adopted to generate masks accordingly. The accuracy of hierarchy is on par with the capability of the model being trained, leading to evolved mask patterns at different training stages. Initially, generated masks focus on low-level visual cues to grasp basic textures, then gradually evolve to depict higher-level cues to reinforce the learning of more complicated object semantics and contexts. Our method does not require extra pre-trained models or annotations and ensures training efficiency by evolving the training difficulty. We conduct extensive experiments on seven downstream tasks including partial-duplicate image retrieval relying on low-level details, as well as image classification and semantic segmentation that require semantic parsing capability. Experimental results demonstrate that it substantially boosts performance across these tasks. For instance, it surpasses the recent MAE by 1.1% in imageNet-1K classification and 1.4% in ADE20K segmentation with the same training epochs. We also align the proposed method with the current research focus on LLMs. The proposed approach bridges the gap with large-scale pre-training on semantic demanding tasks and enhances intricate detail perception in tasks requiring low-level feature recognition.
Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning

Yinda Chen,Wei Huang,Shenglong Zhou,Qi Chen,Zhiwei Xiong

DOI: https://doi.org/10.24963/ijcai.2023/68

2023-10-06

Abstract:The performance of existing supervised neuron segmentation methods is highly dependent on the number of accurate annotations, especially when applied to large scale electron microscopy (EM) data. By extracting semantic information from unlabeled data, self-supervised methods can improve the performance of downstream tasks, among which the mask image model (MIM) has been widely used due to its simplicity and effectiveness in recovering original information from masked images. However, due to the high degree of structural locality in EM images, as well as the existence of considerable noise, many voxels contain little discriminative information, making MIM pretraining inefficient on the neuron segmentation task. To overcome this challenge, we propose a decision-based MIM that utilizes reinforcement learning (RL) to automatically search for optimal image masking ratio and masking strategy. Due to the vast exploration space, using single-agent RL for voxel prediction is impractical. Therefore, we treat each input patch as an agent with a shared behavior policy, allowing for multi-agent collaboration. Furthermore, this multi-agent model can capture dependencies between voxels, which is beneficial for the downstream segmentation task. Experiments conducted on representative EM datasets demonstrate that our approach has a significant advantage over alternative self-supervised methods on the task of neuron segmentation. Code is available at \url{<a class="link-external link-https" href="https://github.com/ydchen0806/dbMiM" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition,Artificial Intelligence
Representation Recovering for Self-Supervised Pre-training on Medical Images.

Xiangyi Yan,Junayed Naushad,Shanlin Sun,Kun Han,Hao Tang,Deying Kong,Haoyu Ma,Chenyu You,Xiaohui Xie

DOI: https://doi.org/10.1109/wacv56688.2023.00271

2023-01-01

Abstract:Advances in self-supervised learning have drawn attention to developing techniques to extract effective visual representations from unlabeled images. Contrastive learning (CL) trains a model to extract consistent features by generating different views. Recent success of Masked Autoencoders (MAE) highlights the benefit of generative modeling in self-supervised learning. The generative approaches encode the input into a compact embedding and empower the model’s ability of recovering the original input. However, in our experiments, we found vanilla MAE mainly recovers coarse high level semantic information and is inadequate in recovering detailed low level information. We show that in dense downstream prediction tasks like multi-organ segmentation, directly applying MAE is not ideal. Here, we propose RepRec, a hybrid visual representation learning framework for self-supervised pre-training on large-scale unlabelled medical datasets, which takes advantage of both contrastive and generative modeling. To solve the aforementioned dilemma that MAE encounters, a convolutional encoder is pre-trained to provide low-level feature information, in a contrastive way; and a transformer encoder is pre-trained to produce high level semantic dependency, in a generative way – by recovering masked representations from the convolutional encoder. Extensive experiments on three multi-organ segmentation datasets demonstrate that our method outperforms current state-of-the-art methods.
Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learning

Yibing Wei,Abhinav Gupta,Pedro Morgado

2024-07-23

Abstract:Masked Image Modeling (MIM) has emerged as a promising method for deriving visual representations from unlabeled image data by predicting missing pixels from masked portions of images. It excels in region-aware learning and provides strong initializations for various tasks, but struggles to capture high-level semantics without further supervised fine-tuning, likely due to the low-level nature of its pixel reconstruction objective. A promising yet unrealized framework is learning representations through masked reconstruction in latent space, combining the locality of MIM with the high-level targets. However, this approach poses significant training challenges as the reconstruction targets are learned in conjunction with the model, potentially leading to trivial or suboptimal solutions.Our study is among the first to thoroughly analyze and address the challenges of such framework, which we refer to as Latent MIM. Through a series of carefully designed experiments and extensive analysis, we identify the source of these challenges, including representation collapsing for joint online/target optimization, learning objectives, the high region correlation in latent space and decoding conditioning. By sequentially addressing these issues, we demonstrate that Latent MIM can indeed learn high-level representations while retaining the benefits of MIM models.

Computer Vision and Pattern Recognition,Artificial Intelligence
Enhancing Semi-Supervised Medical Image Segmentation with Bidirectional Copy-Paste and Masked Image Reconstruction

Xixuan Yu,Qinghe Ma,Tong Ling,Jiewen Zhu,Yinghuan Shi

DOI: https://doi.org/10.1007/s13042-024-02410-1

2024-01-01

International Journal of Machine Learning and Cybernetics

Abstract:To avoid the time-consuming and specialized task of medical image annotation, semi-supervised medical image segmentation methods encourage models to leverage large amounts of unlabeled data for training, which plays a crucial role in real-world medical scenarios. However, existing semi-supervised medical image segmentation methods often rely on data augmentation and pseudo-labeling techniques, overlooking the model’s perception of the relatively fixed structural information in medical images. Consequently, the segmentation results remain suboptimal. We propose integrating the Masked Autoencoders (MAE) method into the Bidirectional Copy-Paste (BCP) framework to address this challenge. Specifically, we randomly mask medical images and introduce an additional image reconstruction task in semi-supervised medical image segmentation to enhance the model encoder’s understanding and perception of image structures. We conducted experiments on the LA dataset and ACDC dataset. Experimental results show that our approach significantly outperforms state-of-the-art semi-supervised medical image segmentation methods, achieving notable improvements on various datasets (e.g., over 1.2 % Dice improvement on ACDC dataset).

Photoperiodic regulation of the diapause of the progeny in Trichogramma embryophagum Htg. (Hymenoptera, Trichogrammatidae): Dynamics of sensitivity to photoperiod at the immature stages of maternal females

AMLP:Adaptive Masking Lesion Patches for Self-supervised Medical Image Segmentation

MPS-AMS: Masked Patches Selection and Adaptive Masking Strategy Based Self-Supervised Medical Image Segmentation

AnatoMask: Enhancing Medical Image Segmentation with Reconstruction-guided Self-masking

When Masked Image Modeling Meets Source-free Unsupervised Domain Adaptation: Dual-Level Masked Network for Semantic Segmentation

HybridMIM: A Hybrid Masked Image Modeling Framework for 3D Medical Image Segmentation

Masked Image Modeling Advances 3D Medical Image Analysis

GMIM: Self-supervised pre-training for 3D medical image segmentation with adaptive and hierarchical masked image modeling

Rethinking masked image modelling for medical image representation

FreMIM: Fourier Transform Meets Masked Image Modeling for Medical Image Segmentation

Masked Image Modeling Boosting Semi-Supervised Semantic Segmentation

FedATA: Adaptive Attention Aggregation for Federated Self-supervised Medical Image Segmentation

Self-Supervised Masking for Unsupervised Anomaly Detection and Localization

MedIM: Boost Medical Image Representation via Radiology Report-Guided Masking

Medical supervised masked autoencoders: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification

Advancing Volumetric Medical Image Segmentation via Global-Local Masked Autoencoder

Evolved Hierarchical Masking for Self-Supervised Learning

Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning

Representation Recovering for Self-Supervised Pre-training on Medical Images.

Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learning

Enhancing Semi-Supervised Medical Image Segmentation with Bidirectional Copy-Paste and Masked Image Reconstruction