InDL: A New Dataset and Benchmark for In-Diagram Logic Interpretation based on Visual Illusion

Haobo Yang,Wenyu Wang,Ze Cao,Zhekai Duan,Xuchen Liu

2023-06-06

Abstract:This paper introduces a novel approach to evaluating deep learning models' capacity for in-diagram logic interpretation. Leveraging the intriguing realm of visual illusions, we establish a unique dataset, InDL, designed to rigorously test and benchmark these models. Deep learning has witnessed remarkable progress in domains such as computer vision and natural language processing. However, models often stumble in tasks requiring logical reasoning due to their inherent 'black box' characteristics, which obscure the decision-making process. Our work presents a new lens to understand these models better by focusing on their handling of visual illusions -- a complex interplay of perception and logic. We utilize six classic geometric optical illusions to create a comparative framework between human and machine visual perception. This methodology offers a quantifiable measure to rank models, elucidating potential weaknesses and providing actionable insights for model improvements. Our experimental results affirm the efficacy of our benchmarking strategy, demonstrating its ability to effectively rank models based on their logic interpretation ability. As part of our commitment to reproducible research, the source code and datasets will be made publicly available at <a class="link-external link-https" href="https://github.com/rabbit-magic-wh/InDL" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the issue of evaluating the capability of deep learning models in logical interpretation within images and uncovering potential weaknesses in these models' logical reasoning through visual illusions. Specifically, the paper proposes a new dataset and benchmarking framework (InDL) to assess the ability of deep learning models to handle visual illusions. Despite significant advancements in fields such as computer vision and natural language processing, the decision-making process of deep learning models is often considered a "black box," particularly underperforming in tasks requiring logical reasoning.

The core objectives of the paper include:
1. Proposing a novel method to evaluate the performance of deep learning models in logical interpretation within images, creating a comparative framework through six classic geometric optical illusions.
2. Ranking different models using quantitative metrics to reveal potential weaknesses and provide actionable insights for model improvement.
3. Exploring the extent to which phenomena observed by humans (such as the perception of visual illusions) are reproduced in deep learning models, thereby deepening the understanding of how AI systems perceive and interpret information.

Through experimental results, the paper validates the effectiveness of the proposed benchmarking strategy and demonstrates the ranking of models in terms of logical interpretation ability, providing directions for further research.