Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions

Akash Ghosh,Arkadeep Acharya,Sriparna Saha,Vinija Jain,Aman Chadha

2024-04-13

Abstract:The advent of Large Language Models (LLMs) has significantly reshaped the trajectory of the AI revolution. Nevertheless, these LLMs exhibit a notable limitation, as they are primarily adept at processing textual information. To address this constraint, researchers have endeavored to integrate visual capabilities with LLMs, resulting in the emergence of Vision-Language Models (VLMs). These advanced models are instrumental in tackling more intricate tasks such as image captioning and visual question answering. In our comprehensive survey paper, we delve into the key advancements within the realm of VLMs. Our classification organizes VLMs into three distinct categories: models dedicated to vision-language understanding, models that process multimodal inputs to generate unimodal (textual) outputs and models that both accept and produce multimodal inputs and outputs.This classification is based on their respective capabilities and functionalities in processing and generating various modalities of data.We meticulously dissect each model, offering an extensive analysis of its foundational architecture, training data sources, as well as its strengths and limitations wherever possible, providing readers with a comprehensive understanding of its essential components. We also analyzed the performance of VLMs in various benchmark datasets. By doing so, we aim to offer a nuanced understanding of the diverse landscape of VLMs. Additionally, we underscore potential avenues for future research in this dynamic domain, anticipating further breakthroughs and advancements.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The main problem that this paper attempts to address is the limitations of existing large language models (LLMs) in handling multimodal data. Specifically, although LLMs perform excellently in processing textual information, they are primarily limited to a single data modality, which is text. To overcome this limitation, researchers are working on integrating visual capabilities with LLMs to develop visual-language models (VLMs). These models are capable of handling complex tasks such as generating image descriptions and answering visual questions.

The paper categorizes existing VLMs into three types through a comprehensive review:
1. **Visual-Language Understanding Models**: Focus on interpreting and understanding the combination of visual information and language.
2. **Multimodal Input Text Generation Models**: Utilize multimodal inputs to generate textual content.
3. **Multimodal Input-Multimodal Output Models**: Handle multimodal inputs and generate multimodal outputs.

Through this classification, the paper provides a detailed analysis of the architecture, training data sources, and the advantages and disadvantages of each type of model, and evaluates their performance on various benchmark datasets. Additionally, the paper explores potential future research directions aimed at further breakthroughs and advancements in this field.