ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense

Kankan Zhou,Eason Lai,Wei Bin Au Yeong,Kyriakos Mouratidis,Jing Jiang

2023-10-30

Abstract:Humans possess a strong capability for reasoning beyond common sense. For example, given an unconventional image of a goldfish laying on the table next to an empty fishbowl, a human would effortlessly determine that the fish is not inside the fishbowl. The case, however, may be different for a vision-language model, whose reasoning could gravitate towards the common scenario that the fish is inside the bowl, despite the visual input. In this paper, we introduce a novel probing dataset named ROME (reasoning beyond commonsense knowledge) to evaluate whether the state-of-the-art pre-trained vision-language models have the reasoning capability to correctly interpret counter-intuitive content. ROME contains images that defy commonsense knowledge with regards to color, shape, material, size and positional relation. Experiments on the state-of-the-art pre-trained vision-language models reveal that most of these models are still largely incapable of interpreting counter-intuitive scenarios. We hope that ROME will spur further investigations on reasoning beyond commonsense knowledge in vision-language research.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate whether the current state - of - the - art pre - trained Vision - Language Models (VLMs) have the ability to go beyond common - sense reasoning. Specifically, the author focuses on the performance of these models in the face of counter - common - sense scenarios. For example, when given an image that does not conform to the normal perception (such as a goldfish is on the table instead of in the fish tank), can the model correctly understand this counter - common - sense content? To evaluate this, the author introduced a new dataset ROM (Reasoning beyond Commonsense Knowledge), which contains a series of images that violate common sense, involving aspects such as color, shape, material, size and positional relationships. Through this dataset, the author hopes to reveal the limitations of current VLMs in dealing with counter - common - sense scenarios and promote further research in this field.