Abstract:Deep neural networks (DNNs) are increasingly proposed as models of human vision, bolstered by their impressive performance on image classification and object recognition tasks. Yet, the extent to which DNNs capture fundamental aspects of human vision such as color perception remains unclear. Here, we develop novel experiments for evaluating the perceptual coherence of color embeddings in DNNs, and we assess how well these algorithms predict human color similarity judgments collected via an online survey. We find that state-of-the-art DNN architectures - including convolutional neural networks and vision transformers - provide color similarity judgments that strikingly diverge from human color judgments of (i) images with controlled color properties, (ii) images generated from online searches, and (iii) real-world images from the canonical CIFAR-10 dataset. We compare DNN performance against an interpretable and cognitively plausible model of color perception based on wavelet decomposition, inspired by foundational theories in computational neuroscience. While one deep learning model - a convolutional DNN trained on a style transfer task - captures some aspects of human color perception, our wavelet algorithm provides more coherent color embeddings that better predict human color judgments compared to all DNNs we examine. These results hold when altering the high-level visual task used to train similar DNN architectures (e.g., image classification versus image segmentation), as well as when examining the color embeddings of different layers in a given DNN architecture. These findings break new ground in the effort to analyze the perceptual representations of machine learning algorithms and to improve their ability to serve as cognitively plausible models of human vision. Implications for machine learning, human perception, and embodied cognition are discussed.

What comparing deep neural networks can teach us about human vision

Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

Comparing object recognition in humans and deep convolutional neural networks -- An eye tracking study

Dimensions underlying the representational alignment of deep neural networks with humans

Using drawings and deep neural networks to characterize the building blocks of human visual similarity

Aligning Machine and Human Visual Representations across Abstraction Levels

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines

A comparison between humans and AI at recognizing objects in unusual poses

Universal dimensions of visual representation

Comparison Against Task Driven Artificial Neural Networks Reveals Functional Organization of Mouse Visual Cortex

Deep neural networks: a new framework for modelling biological vision and brain information processing

Evaluating (and Improving) the Correspondence Between Deep Neural Networks and Human Representations

On the similarities of representations in artificial and brain neural networks for speech recognition

Divergences in color perception between deep neural networks and humans

Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition

Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition

Learning an Adaptation Function to Assess Image Visual Similarities

Evaluating alignment between humans and neural network representations in image-based learning tasks

Deep Spiking Neural Networks with High Representation Similarity Model Visual Pathways of Macaque and Mouse

Deep learning algorithms reveal a new visual-semantic representation of familiar faces in human perception and memory

Convolutional neural networks for vision neuroscience: significance, developments, and outstanding issues