Abstract:Color vision is essential for human visual perception, but its impact on machine perception is still underexplored. There has been an intensified demand for understanding its role in machine perception for safety-critical tasks such as assistive driving and surgery but lacking suitable datasets. To fill this gap, we curate multipurpose datasets ColorSense, by collecting 110,000 non-trivial human annotations of foreground and background color labels from popular visual recognition benchmarks. To investigate the impact of color vision on machine perception, we assign each image a color discrimination level based on its dominant foreground and background colors and use it to study the impact of color vision on machine perception. We validate the use of our datasets by demonstrating that the level of color discrimination has a dominating effect on the performance of mainstream machine perception models. Specifically, we examine the perception ability of machine vision by considering key factors such as model architecture, training objective, model size, training data, and task complexity. Furthermore, to investigate how color and environmental factors affect the robustness of visual recognition in machine perception, we integrate our ColorSense datasets with image corruptions and perform a more comprehensive visual perception evaluation. Our findings suggest that object recognition tasks such as classification and localization are susceptible to color vision bias, especially for high-stakes cases such as vehicle classes, and advanced mitigation techniques such as data augmentation and so on only give marginal improvement. Our analyses highlight the need for new approaches toward the performance evaluation of machine perception models in real-world applications. Lastly, we present various potential applications of ColorSense such as studying spurious correlations.

Color in machine vision and its application

Application Of Color-Encoded Structure Light In 3-D Vision Technology

ColorSense: A Study on Color Vision in Machine Visual Recognition

A New Color Constancy Model for Machine Vision

A Virtual Color Space Based on Physical Model

Research and application of image color feature extraction technology

The color recognition research based on machine vision system

Color in Image and Video Processing: Most Recent Trends and Future Research Directions

Image-Based Quantification of Color and Its Machine Vision and Offline Applications

A Comprehensive Investigation of Color Models used in Image Processing

Color image segmentation: advances and prospects

The mechanism of human color vision and potential implanted devices for artificial color vision

Research on the Technology Used to Inspect the Visual Appearance of Tropical Fruit, based on Machine Vision Color Space

Color Transfer for Images: A Survey

Application of color in vision-based recognition based on shape

Application of Machine Vision Technology in Online Detection of Color Print Quality

On the Color Exertion of the CAI Courseware

Contributions of Shape, Texture, and Color in Visual Recognition

Introducing the Knowledge of Human Vision into the Lecture of Computer Vision

Color reproduction in computer vision

Computer-aided Colorization State-of-the-science: A Survey