Low-Resource Vision Challenges for Foundation Models

Yunhua Zhang,Hazel Doughty,Cees G.M. Snoek

2024-04-12

Abstract:Low-resource settings are well-established in natural language processing, where many languages lack sufficient data for deep learning at scale. However, low-resource problems are under-explored in computer vision. In this paper, we address this gap and explore the challenges of low-resource image tasks with vision foundation models. We first collect a benchmark of genuinely low-resource image data, covering historic maps, circuit diagrams, and mechanical drawings. These low-resource settings all share three challenges: data scarcity, fine-grained differences, and the distribution shift from natural images to the specialized domain of interest. While existing foundation models have shown impressive generalizability, we find they cannot transfer well to our low-resource tasks. To begin to tackle the challenges of low-resource vision, we introduce one simple baseline per challenge. Specifically, we i) enlarge the data space by generative models, ii) adopt the best sub-kernels to encode local regions for fine-grained difference discovery and iii) learn attention for specialized domains. Experiments on our three low-resource tasks demonstrate our proposals already provide a better baseline than transfer learning, data augmentation, and fine-grained methods. This highlights the unique characteristics and challenges of low-resource vision for foundation models that warrant further investigation. Project page:

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper discusses the issues in low-resource computer vision tasks, particularly the challenges faced by base models (e.g. large-scale pre-trained models) in handling data scarcity, fine-grained details, and domain-specific image tasks. The paper proposes a new benchmark dataset that includes low-resource images such as historical maps, circuit diagrams, and mechanical drawings, and identifies three main challenges: data scarcity, fine-grained differences, and distribution shift from natural images to professional domains. The paper also introduces simple baseline methods to address these challenges, including using generative models to expand the data space, encoding local regions through selective subkernel coding to discover fine-grained differences, and learning professional domain attention. Experimental results show that these proposed baseline methods outperform transfer learning, data augmentation, and fine-grained methods, highlighting the uniqueness and challenges of low-resource visual tasks for base models.