Abstract:The advent of large vision-language models (LVLMs) represents a remarkable advance in the quest for artificial general intelligence. However, the model's effectiveness in both specialized and general tasks warrants further investigation. This paper endeavors to evaluate the competency of popular LVLMs in specialized and general tasks, respectively, aiming to offer a comprehensive understanding of these novel models. To gauge their effectiveness in specialized tasks, we employ six challenging tasks in three different application scenarios: natural, healthcare, and industrial. These six tasks include salient/camouflaged/transparent object detection, as well as polyp detection, skin lesion detection, and industrial anomaly detection. We examine the performance of three recent open-source LVLMs, including MiniGPT-v2, LLaVA-1.5, and Shikra, on both visual recognition and localization in these tasks. Moreover, we conduct empirical investigations utilizing the aforementioned LVLMs together with GPT-4V, assessing their multi-modal understanding capabilities in general tasks including object counting, absurd question answering, affordance reasoning, attribute recognition, and spatial relation reasoning. Our investigations reveal that these LVLMs demonstrate limited proficiency not only in specialized tasks but also in general tasks. We delve deep into this inadequacy and uncover several potential factors, including limited cognition in specialized tasks, object hallucination, text-to-image interference, and decreased robustness in complex problems. We hope that this study can provide useful insights for the future development of LVLMs, helping researchers improve LVLMs for both general and specialized applications.

Do large language vision models understand 3D shapes?

Do Vision-Language Models Really Understand Visual Language?

Can 3D Vision-Language Models Truly Understand Natural Language?

Vision language models are blind

Language grounding with 3d objects

VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information

Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models

Can Language Models Understand Physical Concepts?

Visual cognition in multimodal large language models

Language-Image Models with 3D Understanding

Effectiveness Assessment of Recent Large Vision-Language Models

ShapeGlot: Learning Language for Shape Differentiation

Are VLMs Really Blind

Do Multimodal Large Language Models See Like Humans?

MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?

Do Vision and Language Models Share Concepts? A Vector Space Alignment Study

RelationVLM: Making Large Vision-Language Models Understand Visual Relations

Large Language Models Understand Layout

A Vision Check-up for Language Models

A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?