Abstract:Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their efficacy. This paper presents a comprehensive evaluation of publicly available large multimodal models by building a LVLM evaluation Hub (LVLM-eHub). Our LVLM-eHub consists of $8$ representative LVLMs such as InstructBLIP and MiniGPT-4, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The former evaluates $6$ categories of multimodal capabilities of LVLMs such as visual question answering and embodied artificial intelligence on $47$ standard text-related visual benchmarks, while the latter provides the user-level evaluation of LVLMs in an open-world question-answering scenario. The study reveals several innovative findings. First, instruction-tuned LVLM with massive in-domain data such as InstructBLIP heavily overfits many existing tasks, generalizing poorly in the open-world scenario. Second, instruction-tuned LVLM with moderate instruction-following data may result in object hallucination issues (i.e., generate objects that are inconsistent with target images in the descriptions). It either makes the current evaluation metric such as CIDEr for image captioning ineffective or generates wrong answers. Third, employing a multi-turn reasoning evaluation framework can mitigate the issue of object hallucination, shedding light on developing an effective pipeline for LVLM evaluation. The findings provide a foundational framework for the conception and assessment of innovative strategies aimed at enhancing zero-shot multimodal techniques. Our LVLM-eHub will be available at <a class="link-external link-https" href="https://github.com/OpenGVLab/Multi-Modality-Arena" rel="external noopener nofollow">this https URL</a>

Collaborative Training of Tiny-Large Vision Language Models

TinyLLaVA: A Framework of Small-scale Large Multimodal Models

InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning

LM4LV: A Frozen Large Language Model for Low-level Vision Tasks

LLaVA-KD: A Framework of Distilling Multimodal Large Language Models

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Unified Generative and Discriminative Training for Multi-modal Large Language Models

Enhancing Advanced Visual Reasoning Ability of Large Language Models

Cross-model Control: Improving Multiple Large Language Models in One-time Training

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model

Small Language Model Meets with Reinforced Vision Vocabulary

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

InfMLLM: A Unified Framework for Visual-Language Tasks.

RelationVLM: Making Large Vision-Language Models Understand Visual Relations

Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference