Abstract:Agglomerative models have recently emerged as a powerful approach to training vision foundation models, leveraging multi-teacher distillation from existing models such as CLIP, DINO, and SAM. This strategy enables the efficient creation of robust models, combining the strengths of individual teachers while significantly reducing computational and resource demands. In this paper, we thoroughly analyze state-of-the-art agglomerative models, identifying critical challenges including resolution mode shifts, teacher imbalance, idiosyncratic teacher artifacts, and an excessive number of output tokens. To address these issues, we propose several novel solutions: multi-resolution training, mosaic augmentation, and improved balancing of teacher loss functions. Specifically, in the context of Vision Language Models, we introduce a token compression technique to maintain high-resolution information within a fixed token count. We release our top-performing models, available in multiple scales (-B, -L, -H, and -g), alongside inference code and pretrained weights.

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly include the following points: 1. **Resolution balance**: Due to differences in architecture and training objectives, existing teacher models operate at different resolutions, resulting in inconsistent feature granularity. Effective techniques are required to balance these resolutions in the student model in order to capture fine - grained details and broader abstract information. 2. **Teacher distribution balance**: Existing models have different distribution characteristics, and the distillation process should take this into account to prevent learning bias. 3. **Generate multi - resolution features to support diverse applications**: Visual models support multiple applications and require different feature resolutions, such as image captioning and dense segmentation. A visual foundation model (VFM) that can flexibly generate features of any resolution can serve multiple tasks, reducing the need for separate models and opening up new opportunities. Specifically, the key challenges mentioned in the paper include: - **Mode - switching phenomenon**: In AM - RADIO, the feature distribution changes significantly with the change of input resolution. Low - resolution inputs produce DINO - like features, and high - resolution inputs produce SAM - like features. This behavior is because the student has learned different features from teachers at different resolutions during the training process. - **The number of tokens in visual - language models**: When dealing with high - resolution images, most methods will return more image tokens, resulting in a quadratic increase in the complexity of the attention mechanism. In order to retain high - resolution information while reducing the number of tokens, the paper proposes a token compression technique. To solve these problems, the paper proposes the following solutions: - **Multi - resolution training**: Through the multi - resolution training strategy, fix the accuracy of the student model at all resolutions. - **Mosaic augmentation**: Improve training efficiency, reduce the cost of learning from high - resolution teachers, and eliminate the need for feature interpolation. - **Improved teacher loss function balance**: Adopt the PCA - Hadamard Isotropic Standardization (PHI - S) method to balance the teacher loss function and improve training stability and overall benchmark performance. - **Token compression**: Use the bilinear soft - matching technique to merge similar tokens, retain the most diverse information, and maintain performance even at a high compression ratio. Through these methods, the paper aims to improve the robustness and performance of visual foundation models at different resolutions, especially in the application of visual - language models.

RADIO Amplified: Improved Baselines for Agglomerative Vision Foundation Models

AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One

POINTS: Improving Your Vision-language Model with Affordable Strategies

Improved Baselines with Visual Instruction Tuning

Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models

EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning

AMD: Automatic Multi-step Distillation of Large-scale Vision Models

TokenUnify: Scalable Autoregressive Visual Pre-training with Mixture Token Prediction

Accessing Vision Foundation Models at ImageNet-level Costs

InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

VILA$^2$: VILA Augmented VILA

Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models

Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks

Rethinking Overlooked Aspects in Vision-Language Models

MVFormer: Diversifying Feature Normalization and Token Mixing for Efficient Vision Transformers

ViM: Vision Middleware for Unified Downstream Transferring

Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities

Distilling Large Vision-Language Model with Out-of-Distribution Generalizability

Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Models