Abstract:In recent years, zero-shot and few-shot learning in visual grounding have garnered considerable attention, largely due to the success of large-scale vision-language pre-training on expansive datasets such as LAION-5B and DataComp-1B. However, the continuous expansion of these datasets presents significant challenges, particularly with respect to data availability and computational overhead, thus creating a bottleneck in the advancement of low-shot learning capabilities. In this paper, we propose IMAGE, Interpretative MAsking with Gaussian radiation modEling, aimed at enhancing vocabulary grounding in low-shot learning scenarios without necessitating an increase in dataset size. Drawing inspiration from cognitive science and the recent success of masked autoencoders (MAE), our method leverages adaptive masking on salient regions of the feature maps generated by the vision backbone. This enables the model to learn robust, generalized representations through the reconstruction of occluded information, thereby facilitating effective attention to both local and global features. We evaluate the efficacy of our approach on benchmark datasets, including COCO and ODinW, demonstrating its superior performance in zero-shot and few-shot tasks. Experimental results consistently show that IMAGE outperforms baseline models, achieving enhanced generalization and improved performance in low-shot scenarios. These findings highlight the potential of adaptive feature manipulation through attention mechanisms and Gaussian modeling as a promising alternative to approaches that rely on the continual scaling of dataset sizes for the advancement of zero-shot and few-shot learning. Our code is publicly available at <a class="link-external link-https" href="https://github.com/git-lenny/IMAGE" rel="external noopener nofollow">this https URL</a>.

Learning to Improve Out-of-Distribution Generalization Via Self-adaptive Language Masking

Masked Autoencoders for Point Cloud Self-supervised Learning.

TLM: Token-Level Masking for Transformers

Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models

Improving the Reusability of Pre-trained Language Models in Real-world Applications

Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Models

Pretrained Transformers Improve Out-of-Distribution Robustness

Out-of-distribution generalization via composition: a lens through induction heads in Transformers

Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model

Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting

Train No Evil: Selective Masking for Task-Guided Pre-Training

Gradient Sparsification For Masked Fine-Tuning of Transformers

Masked Autoencoding for Scalable and Generalizable Decision Making

AMOM: Adaptive Masking over Masking for Conditional Masked Language Model

Self-Evolution Learning for Discriminative Language Model Pretraining.

Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capability

Adaptive Masking Enhances Visual Grounding

Variance-reduced Language Pretraining via a Mask Proposal Network

A Simple yet Effective Self-Debiasing Framework for Transformer Models

How Transformers Learn Diverse Attention Correlations in Masked Vision Pretraining

Masked Images Are Counterfactual Samples for Robust Fine-tuning