Abstract:Few-shot open-set recognition, as a new paradigm, leveraging a limited amount of supervised data to identify specific Remote Sensing (RS) scene categories and generalize to novel ones. However, the data bias induced by the small sample size not only causes severe overfitting within base classes, but also impairs the capacity for inference to identify RS scenes in hitherto unobserved categories. Furthermore, owing to environmental influences, RS images frequently manifest notable intra-class disparities and comparatively low inter-class distinctions, intensifying the challenge in obtaining suitable classifiers. To address above issues, we investigate the utilization of a Multi-modal Foundational Model (MFM) infused with essential domain knowledge to mitigate the generalization limitations encountered in few-shot scenarios. Recognizing that existing MFMs with a visual-text dual-branch structure are primarily tailored for natural scenes, we propose a custom Frequency Distribution-based Multi-modal Fine-Tuning strategy (FreqDiMFT) in a parameter-efficient manner. More specifically, within the vision branch, we address the high inter-class similarity and intra-class diversity in RS images by embedding the local-global frequency distribution information to facilitate the recognition of RS scenes. To further amplify the model's generalization ability post transfer, we introduce an adaptive feature refinement module designed for Transformers, proficient in filtering redundant features resulting from domain disparities. To mitigate the domain drift on the textual branch, we adopt an input format that combines basic templates with domain expertise from RS end to generate more discriminative class prototypes. To fully verify the effectiveness of our FreqDiMFT in a more practical setting, we collect a Large-Scale hybrid dataset (LSRS). Extensive experiments demonstrate that, even with a scant number of training samples, our strategy yields advanced performances compared to state-of-the-art models.

Foundation Model-Based Multimodal Remote Sensing Data Classification

When are Foundation Models Effective? Understanding the Suitability for Pixel-Level Classification Using Multispectral Imagery

More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

Novel Cross-Resolution Feature-Level Fusion for Joint Classification of Multispectral and Panchromatic Remote Sensing Images

Multimodal Foundation Models: From Specialists to General-Purpose Assistants

Promoting cross-modal representations to improve multimodal foundation models for physiological signals

SimCMF: A Simple Cross-modal Fine-tuning Strategy from Vision Foundation Models to Any Imaging Modality

On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence

Towards a Knowledge guided Multimodal Foundation Model for Spatio-Temporal Remote Sensing Applications

A Unified Multimodal Deep Learning Framework for Remote Sensing Imagery Classification.

Remote Sensing Collaborative Classification Using Multimodal Adaptive Modulation Network

Foundation Models for Remote Sensing and Earth Observation: A Survey

MedFMC: A Real-world Dataset and Benchmark For Foundation Model Adaptation in Medical Image Classification

Continual Learning Meets Multimodal Foundation Models: Fundamentals and Advances

Brain-Inspired Remote Sensing Foundation Models and Open Problems: A Comprehensive Survey

MM-UrbanFAC: Urban Functional Area Classification Model Based on Multimodal Machine Learning

A New Learning Paradigm for Foundation Model-Based Remote-Sensing Change Detection

Generative ConvNet Foundation Model With Sparse Modeling and Low-Frequency Reconstruction for Remote Sensing Image Interpretation

Frequency-Aware Multi-Modal Fine-Tuning for Few-Shot Open-Set Remote Sensing Scene Classification

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval