BRAVE: Broadening the visual encoding of vision-language models

Oğuzhan Fatih Kar,Alessio Tonioni,Petra Poklukar,Achin Kulshrestha,Amir Zamir,Federico Tombari
2024-04-11
Abstract:Vision-language models (VLMs) are typically composed of a vision encoder, e.g. CLIP, and a language model (LM) that interprets the encoded features to solve downstream tasks. Despite remarkable progress, VLMs are subject to several shortcomings due to the limited capabilities of vision encoders, e.g. "blindness" to certain image features, visual hallucination, etc. To address these issues, we study broadening the visual encoding capabilities of VLMs. We first comprehensively benchmark several vision encoders with different inductive biases for solving VLM tasks. We observe that there is no single encoding configuration that consistently achieves top performance across different tasks, and encoders with different biases can perform surprisingly similarly. Motivated by this, we introduce a method, named BRAVE, that consolidates features from multiple frozen encoders into a more versatile representation that can be directly fed as the input to a frozen LM. BRAVE achieves state-of-the-art performance on a broad range of captioning and VQA benchmarks and significantly reduces the aforementioned issues of VLMs, while requiring a smaller number of trainable parameters than existing methods and having a more compressed representation. Our results highlight the potential of incorporating different visual biases for a more broad and contextualized visual understanding of VLMs.
Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
This paper focuses on enhancing the visual capabilities of Vision-Language Models (VLMs). Existing VLMs typically consist of a visual encoder, such as CLIP, and a language model, but they have limitations in handling downstream tasks, such as "blindness" to certain image features and visual illusions. To overcome these issues, the paper proposes a method called BRAVE, which expands the visual encoding of VLMs. BRAVE creates a more flexible and compressed representation by combining features from multiple visual encoders with different priors, which can be directly input into the frozen language model. It was found that different visual encoders perform differently on various visual language tasks, and no encoding configuration can consistently achieve optimal performance. Therefore, BRAVE integrates these features using a Multi-Encoder Query Transformer (MEQ-Former), improving the model's performance in a wide range of tasks, including image captioning and visual question answering, while significantly reducing the number of training parameters. Experimental results show that BRAVE achieves state-of-the-art performance on multiple standard image captioning and visual question answering tasks, while improving robustness to out-of-distribution inputs and reducing visual illusion issues. This indicates that by merging different visual biases, VLMs can enhance their comprehensive and contextual understanding of visual scenes.