Abstract:Color vision is essential for human visual perception, but its impact on machine perception is still underexplored. There has been an intensified demand for understanding its role in machine perception for safety-critical tasks such as assistive driving and surgery but lacking suitable datasets. To fill this gap, we curate multipurpose datasets ColorSense, by collecting 110,000 non-trivial human annotations of foreground and background color labels from popular visual recognition benchmarks. To investigate the impact of color vision on machine perception, we assign each image a color discrimination level based on its dominant foreground and background colors and use it to study the impact of color vision on machine perception. We validate the use of our datasets by demonstrating that the level of color discrimination has a dominating effect on the performance of mainstream machine perception models. Specifically, we examine the perception ability of machine vision by considering key factors such as model architecture, training objective, model size, training data, and task complexity. Furthermore, to investigate how color and environmental factors affect the robustness of visual recognition in machine perception, we integrate our ColorSense datasets with image corruptions and perform a more comprehensive visual perception evaluation. Our findings suggest that object recognition tasks such as classification and localization are susceptible to color vision bias, especially for high-stakes cases such as vehicle classes, and advanced mitigation techniques such as data augmentation and so on only give marginal improvement. Our analyses highlight the need for new approaches toward the performance evaluation of machine perception models in real-world applications. Lastly, we present various potential applications of ColorSense such as studying spurious correlations.

Evaluating Model Perception of Color Illusions in Photorealistic Scenes

Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?

Investigating Color Illusions from the Perspective of Computational Color Constancy

IllusionBench: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models

Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions

IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

The Illusion-Illusion: Vision Language Models See Illusions Where There are None

The Art of Deception: Color Visual Illusions and Diffusion Models

Image Visual Realism: from Human Perception to Machine Computation.

ColorSense: A Study on Color Vision in Machine Visual Recognition

Towards Photorealistic Colorization by Imagination

Refining Skewed Perceptions in Vision-Language Models through Visual Representations

Deep learning models for perception of brightness related illusions

Visual Illusions Also Deceive Convolutional Neural Networks: Analysis and Implications

Vision-Language Models under Cultural and Inclusive Considerations

Computer Vision Datasets and Models Exhibit Cultural and Linguistic Diversity in Perception

A Retinal Mechanism Inspired Color Constancy Model

Thinking Image Color Aesthetics Assessment: Models, Datasets and Benchmarks

BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models

Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models

The Instinctive Bias: Spurious Images lead to Illusion in MLLMs