Abstract:Scaling up the size of vision models has been the de facto standard to obtain more powerful visual representations. In this work, we discuss the point beyond which larger vision models are not necessary. First, we demonstrate the power of Scaling on Scales (S$^2$), whereby a pre-trained and frozen smaller vision model (e.g., ViT-B or ViT-L), run over multiple image scales, can outperform larger models (e.g., ViT-H or ViT-G) on classification, segmentation, depth estimation, Multimodal LLM (MLLM) benchmarks, and robotic manipulation. Notably, S$^2$ achieves state-of-the-art performance in detailed understanding of MLLM on the V* benchmark, surpassing models such as GPT-4V. We examine the conditions under which S$^2$ is a preferred scaling approach compared to scaling on model size. While larger models have the advantage of better generalization on hard examples, we show that features of larger vision models can be well approximated by those of multi-scale smaller models. This suggests most, if not all, of the representations learned by current large pre-trained models can also be obtained from multi-scale smaller models. Our results show that a multi-scale smaller model has comparable learning capacity to a larger model, and pre-training smaller models with S$^2$ can match or even exceed the advantage of larger models. We release a Python package that can apply S$^2$ on any vision model with one line of code: <a class="link-external link-https" href="https://github.com/bfshi/scaling_on_scales" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper explores whether larger models are always necessary to achieve better performance in visual understanding tasks. Specifically, the authors propose a method called "Scaling on Scales" (S2), which improves model performance by running pre-trained small visual models on multiple image scales, rather than simply increasing the model's parameter count. ### Main Findings 1. **Effectiveness of the S2 Method**: - The authors demonstrate that in tasks such as classification, semantic segmentation, depth estimation, multi-modal large language model (MLLM) benchmarks, and robotic operations, the S2 method enables small models (e.g., ViT-B or ViT-L) to outperform large models (e.g., ViT-H or ViT-G) with significantly fewer parameters (0.28x to 0.07x) and comparable computational complexity (GFLOPS). - Particularly in the visual detail understanding tasks of MLLM, by scaling the image size up to 1008², the S2 method achieved state-of-the-art performance in the V* benchmark, surpassing both open-source and commercial MLLM models such as Gemini Pro and GPT-4V. 2. **Comparison of S2 and Model Size Scaling**: - The authors experimentally compared the performance of the S2 method and model size scaling across different tasks. The results show that in most cases, the S2 method can achieve or exceed the performance of large models with the same computational resources. - However, for certain specific tasks (e.g., rare and blurry samples in image classification), large models still exhibit better generalization capabilities. 3. **Conditions for the Applicability of the S2 Method**: - The authors further investigated the conditions under which the S2 method outperforms model size scaling. They found that while the S2 method can achieve better downstream task performance in many scenarios, large models still show stronger generalization capabilities when dealing with difficult samples. - Through experiments, the authors discovered that multi-scale small models can approximate the features of large models well through a single linear transformation, indicating that small models have similar learning capabilities to large models. They hypothesize that the weak generalization ability of small models may be due to their pre-training on a single image scale. ### Conclusion This paper proposes the S2 method as an effective alternative that can enhance the performance of visual models through multi-scale image processing without increasing the model's parameter count. Although large models still have advantages in certain specific tasks, the S2 method can achieve or exceed the performance of large models in most cases with lower computational resource requirements. This provides new insights for optimizing visual models.

When Do We Not Need Larger Vision Models?

Scaling Vision Transformers

SLViT: Scale-Wise Language-Guided Vision Transformer for Referring Image Segmentation.

Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

Revisiting Neural Scaling Laws in Language and Vision

Are Bigger Encoders Always Better in Vision Large Models?

Applicability of scaling laws to vision encoding models

Scaling may be all you need for achieving human-level object recognition capacity with human-like visual experience

Slicing Vision Transformer for Flexible Inference

Inverse Scaling: When Bigger Isn't Better

Sequential Modeling Enables Scalable Learning for Large Vision Models

Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model

Scaling White-Box Transformers for Vision

Scaling Laws Do Not Scale

ScopeViT: Scale-aware Vision Transformer

Towards Precise Scaling Laws for Video Diffusion Transformers

VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models

Auto-scaling Vision Transformers without Training

What matters when building vision-language models?