Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

Leonard E. van Dyck,Walter R. Gruber

DOI: https://doi.org/10.48550/arXiv.2007.06294

2020-12-13

Abstract:For a considerable time, deep convolutional neural networks (DCNNs) have reached human benchmark performance in object recognition. On that account, computational neuroscience and the field of machine learning have started to attribute numerous similarities and differences to artificial and biological vision. This study aims towards a behavioral comparison of visual core object recognition performance between humans and feedforward neural networks in a classification learning paradigm on an ImageNet data set. For this purpose, human participants (n = 65) competed in an online experiment against different feedforward DCNNs. The designed approach based on a typical learning process of seven different monkey categories included a training and validation phase with natural examples, as well as a testing phase with novel, unexperienced shape and color manipulations. Analyses of accuracy revealed that humans not only outperform DCNNs on all conditions, but also display significantly greater robustness towards shape and most notably color alterations. Furthermore, a precise examination of behavioral patterns highlights these findings by revealing independent classification errors between the groups. The obtained results show that humans contrast strongly with artificial feedforward architectures when it comes to visual core object recognition of manipulated images. In general, these findings are in line with a growing body of literature, that hints towards recurrence as a crucial factor for adequate generalization abilities.

Computer Vision and Pattern Recognition,Machine Learning,Image and Video Processing,Neurons and Cognition

What problem does this paper attempt to address?

The paper attempts to address the differences in object recognition performance between humans and deep convolutional neural networks (DCNNs) under image manipulation conditions. Specifically, the researchers explored the following questions by comparing the performance of humans and several feedforward deep convolutional neural networks on classification tasks: 1. **What are the differences in recognition accuracy between humans and DCNNs on natural images and images manipulated by shape or color?** 2. **How robust are humans and DCNNs when faced with shape and color manipulations?** 3. **Are there significant differences in the patterns of classification errors between humans and DCNNs?** The researchers designed an experiment with training, validation, and testing phases, where human participants and three different DCNNs (AlexNet, GoogLeNet, and ResNet-50) competed on the same dataset. The dataset included images of 7 different types of monkeys, and the testing phase included challenge images with shape and color manipulations. By analyzing accuracy and confusion matrices, the researchers aimed to reveal the similarities and differences between the human visual system and artificial neural networks in core object recognition tasks. The main conclusions include: - Humans outperformed DCNNs under all conditions, especially on images with shape and color manipulations. - Humans showed significantly higher robustness to color manipulations compared to DCNNs. - There were significant differences in the patterns of classification errors between humans and DCNNs, indicating that they employ different strategies when processing complex visual information. These findings support the importance of recursive mechanisms in biological visual systems and suggest that current feedforward DCNNs have limitations in handling atypical images.

Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

Comparing object recognition in humans and deep convolutional neural networks -- An eye tracking study

Improved object recognition using neural networks trained to mimic the brain's statistical properties

Humans and deep networks largely agree on which kinds of variation make object recognition harder

Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition

A comparison between humans and AI at recognizing objects in unusual poses

Guiding Visual Attention in Deep Convolutional Neural Networks Based on Human Eye Movements

Human Visual Cortex and Deep Convolutional Neural Network Care Deeply about Object Background

Evaluating (and Improving) the Correspondence Between Deep Neural Networks and Human Representations

Divergences in color perception between deep neural networks and humans

What comparing deep neural networks can teach us about human vision

Crowding in humans is unlike that in convolutional neural networks

Partial success in closing the gap between human and machine vision

Face recognition depends on specialized mechanisms tuned to view-invariant facial features: Insights from deep neural networks optimized for face or object recognition

Equivalent processing of facial expression and identity by macaque visual system and task-optimized neural network

Comparison Against Task Driven Artificial Neural Networks Reveals Functional Organization of Mouse Visual Cortex

Deep Convolutional Neural Networks Outperform Feature-Based But Not Categorical Models in Explaining Object Similarity Judgments

Modeling Biological Face Recognition with Deep Convolutional Neural Networks

Manipulating and measuring variation in deep neural network (DNN) representations of objects

Twin identification over viewpoint change: A deep convolutional neural network surpasses humans

Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex