Abstract:Deep neural networks (DNNs) are increasingly proposed as models of human vision, bolstered by their impressive performance on image classification and object recognition tasks. Yet, the extent to which DNNs capture fundamental aspects of human vision such as color perception remains unclear. Here, we develop novel experiments for evaluating the perceptual coherence of color embeddings in DNNs, and we assess how well these algorithms predict human color similarity judgments collected via an online survey. We find that state-of-the-art DNN architectures - including convolutional neural networks and vision transformers - provide color similarity judgments that strikingly diverge from human color judgments of (i) images with controlled color properties, (ii) images generated from online searches, and (iii) real-world images from the canonical CIFAR-10 dataset. We compare DNN performance against an interpretable and cognitively plausible model of color perception based on wavelet decomposition, inspired by foundational theories in computational neuroscience. While one deep learning model - a convolutional DNN trained on a style transfer task - captures some aspects of human color perception, our wavelet algorithm provides more coherent color embeddings that better predict human color judgments compared to all DNNs we examine. These results hold when altering the high-level visual task used to train similar DNN architectures (e.g., image classification versus image segmentation), as well as when examining the color embeddings of different layers in a given DNN architecture. These findings break new ground in the effort to analyze the perceptual representations of machine learning algorithms and to improve their ability to serve as cognitively plausible models of human vision. Implications for machine learning, human perception, and embodied cognition are discussed.

Spatial and Colour Opponency in Anatomically Constrained Deep Networks

A neural field model for color perception unifying assimilation and contrast

Coding of chromatic spatial contrast by macaque V1 neurons

A Neurocomputing Model for Ganglion Cell's Color Opponency Mechanism and Its Application in Image Analysis

Divergences in color perception between deep neural networks and humans

Exploring the Categorical Nature of Colour Perception: Insights from Artificial Networks

Improved Color Opponent Contour Detection Model Based on Dark and Light Adaptation.

A Neuronal Network Model of the Primate Visual System: Color Mechanisms in the Retina, LGN and V1

ColorNet: Investigating the importance of color spaces for image classification

Learning Color Equivariant Representations

Color Constancy Using Double-Opponency.

Hierarchical binding in convolutional neural networks: Making adversarial attacks geometrically challenging

CHARACTERISING REPRESENTATIONS OF HUE AND SATURATION IN THE CORTEX USING INFORMATION DECODING

Boundary Detection Using Double-Opponency and Spatial Sparseness Constraint

Color Equivariant Convolutional Networks

Saliency Suppressed, Semantics Surfaced: Visual Transformations in Neural Networks and the Brain

Dissociable Neural Representations of Adversarially Perturbed Images in Convolutional Neural Networks and the Human Brain

A Computational Framework for Modeling Emergence of Color Vision in the Human Brain

Covariant spatio-temporal receptive fields for neuromorphic computing

A Color-Opponency Based Biological Model for Color Constancy