Abstract:Deep neural networks (DNNs) are increasingly proposed as models of human vision, bolstered by their impressive performance on image classification and object recognition tasks. Yet, the extent to which DNNs capture fundamental aspects of human vision such as color perception remains unclear. Here, we develop novel experiments for evaluating the perceptual coherence of color embeddings in DNNs, and we assess how well these algorithms predict human color similarity judgments collected via an online survey. We find that state-of-the-art DNN architectures - including convolutional neural networks and vision transformers - provide color similarity judgments that strikingly diverge from human color judgments of (i) images with controlled color properties, (ii) images generated from online searches, and (iii) real-world images from the canonical CIFAR-10 dataset. We compare DNN performance against an interpretable and cognitively plausible model of color perception based on wavelet decomposition, inspired by foundational theories in computational neuroscience. While one deep learning model - a convolutional DNN trained on a style transfer task - captures some aspects of human color perception, our wavelet algorithm provides more coherent color embeddings that better predict human color judgments compared to all DNNs we examine. These results hold when altering the high-level visual task used to train similar DNN architectures (e.g., image classification versus image segmentation), as well as when examining the color embeddings of different layers in a given DNN architecture. These findings break new ground in the effort to analyze the perceptual representations of machine learning algorithms and to improve their ability to serve as cognitively plausible models of human vision. Implications for machine learning, human perception, and embodied cognition are discussed.

Neither hype nor gloom do DNNs justice

Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?

Deep Nets: What have they ever done for Vision?

Fixing the problems of deep neural networks will require better training data and learning algorithms

Intriguing properties of neural networks

Shallow-Deep Networks: Understanding and Mitigating Network Overthinking

Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

An Overview of Convergence Analysis of Deep Neural Networks and the Visual Explanations of their Predictions

Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex

Visual Analysis of Prediction Uncertainty in Neural Networks for Deep Image Synthesis

DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanations

Deep Learning Opacity in Scientific Discovery

Library network, a possible path to explainable neural networks

A neural network walks into a lab: towards using deep nets as models for human behavior

Divergences in color perception between deep neural networks and humans

Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications

Deep Learning: The Good, the Bad, and the Ugly

Leveraging the HW/SW Optimizations and Ecosystems that Drive the AI Revolution

A Probabilistic Theory of Deep Learning

Do DNNs trained on Natural Images acquire Gestalt Properties?

Understanding Neural Networks Through Deep Visualization