Abstract:In recent years, considerable research has been conducted on vision-language models that handle both image and text data; these models are being applied to diverse downstream tasks, such as "image-related chat," "image recognition by instruction," and "answering visual questions." Vision-language models (VLMs), such as Contrastive Language-Image Pre-training (CLIP), are also high-performance image classifiers that are being developed into domain adaptation methods that can utilize language information to extend into unseen domains. However, because these VLMs embed images as a single point in a unified embedding space, there is room for improvement in the classification accuracy. Therefore, in this study, we proposed the Latent Augmentation using Regional Embedding (LARE), which embeds the image as a region in the unified embedding space learned by the VLM. By sampling the augmented image embeddings from within this latent region, LARE enables data augmentation to various unseen domains, not just to specific unseen domains. LARE achieves robust image classification for domains in and out using augmented image embeddings to fine-tune VLMs. We demonstrate that LARE outperforms previous fine-tuning models in terms of image classification accuracy on three benchmarks. We also demonstrate that LARE is a more robust and general model that is valid under multiple conditions, such as unseen domains, small amounts of data, and imbalanced data.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is the insufficient classification accuracy of existing Vision - Language Models (VLMs) when handling image classification tasks, which is caused by embedding images as a single point in a unified embedding space. Specifically:
1. **Limitations of existing VLMs**: Existing VLMs (such as CLIP) perform multi - modal tasks by embedding images and texts into a unified space. However, in image classification tasks, they represent images as a single point, which limits their generalization ability in unseen domains.
2. **Challenges in data augmentation**: In order to improve performance in unseen domains, generative models (such as Stable Diffusion or DALL - E) are usually used to generate synthetic images for data augmentation. However, these methods cannot faithfully follow the instructions of specific tasks and may generate noisy images that are irrelevant to the tasks, thus affecting the performance of downstream tasks.
3. **The need for domain adaptation**: Effective domain adaptation in unseen domains is crucial. Although existing methods can enhance data in specific unseen domains, they lack consideration of the diversity of various domains in the test set and are prone to over - fitting to specific domains.
For this reason, the authors propose the **Latent Augmentation using Regional Embedding (LARE)** method, which aims to solve the above problems in the following ways:
- **Regional Embedding**: LARE embeds an image into a region (rather than a single point) in the unified embedding space and enhances the image embedding by sampling from this region. This method can not only enhance various unseen domains but also retain the category information of the original image.
- **Robustness and generalization ability**: By introducing regional embedding in the training process, LARE can build a more robust and general - purpose image classification model, which is suitable for image classification tasks under various conditions, including unseen domains, small - amount data, and imbalanced data.
### Specific implementation
The implementation of LARE is divided into two stages:
1. **Stage 1: Learning the region (Box)**
- Train a neural network \( f_{\text{Box}}: \mathbb{R}^d \to \mathbb{R}^{2d} \) to convert each image embedding into a region (box) in the latent space. This region is defined by two corner points \( X^- \) and \( X^+ \), ensuring that the region is large enough to contain unseen domains while retaining the category information of the original image.
- Use Box Volume Loss and Class Consistency Loss to optimize \( f_{\text{Box}} \), ensuring that the region size is appropriate and the category consistency is good.
2. **Stage 2: Fine - tuning**
- Fine - tune the VLM on a training set containing the original image embeddings and the augmented image embeddings randomly sampled from the region, and train a linear classifier using the linear probing technique.
- In this way, LARE builds a more robust image classification model that can perform better in various unseen domains.
### Experimental results
The experimental results show that LARE significantly outperforms other methods in image classification tasks on multiple benchmark datasets (such as CUB, DomainNet, and CIFAR - 100), especially in unseen domains, few - shot learning, and imbalanced data conditions.
In summary, LARE proposes a new data augmentation method, which improves the robustness and generalization ability of VLM in image classification tasks through regional embedding.