Abstract:Deep networks should be robust to rare events if they are to be successfully deployed in high-stakes real-world applications (e.g., self-driving cars). Here we study the capability of deep networks to recognize objects in unusual poses. We create a synthetic dataset of images of objects in unusual orientations, and evaluate the robustness of a collection of 38 recent and competitive deep networks for image classification. We show that classifying these images is still a challenge for all networks tested, with an average accuracy drop of 29.5% compared to when the objects are presented upright. This brittleness is largely unaffected by various network design choices, such as training losses (e.g., supervised vs. self-supervised), architectures (e.g., convolutional networks vs. transformers), dataset modalities (e.g., images vs. image-text pairs), and data-augmentation schemes. However, networks trained on very large datasets substantially outperform others, with the best network tested$\unicode{x2014}$Noisy Student EfficentNet-L2 trained on JFT-300M$\unicode{x2014}$showing a relatively small accuracy drop of only 14.5% on unusual poses. Nevertheless, a visual inspection of the failures of Noisy Student reveals a remaining gap in robustness with the human visual system. Furthermore, combining multiple object transformations$\unicode{x2014}$3D-rotations and scaling$\unicode{x2014}$further degrades the performance of all networks. Altogether, our results provide another measurement of the robustness of deep networks that is important to consider when using them in the real world. Code and datasets are available at <a class="link-external link-https" href="https://github.com/amro-kamal/ObjectPose" rel="external noopener nofollow">this https URL</a>.

On the Performance of GoogLeNet and AlexNet Applied to Sketches

Sketch-R2CNN: an Attentive Network for Vector Sketch Recognition

<i>Sketch-R2CNN</i>: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

On Learning Semantic Representations for Large-Scale Abstract Sketches

Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

Do Generalised Classifiers really work on Human Drawn Sketches?

25 years of CNNs: Can we compare to human abstraction capabilities?

SSR-GNNs: Stroke-based Sketch Representation with Graph Neural Networks

The Performance of Two CNN Methods in Artworks Aesthetic Feature Recognition

SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

Visualizing and Understanding Convolutional Networks

Progress and limitations of deep networks to recognize objects in unusual poses

Learning Abstract Classes using Deep Learning

SketchGCN: Semantic Sketch Segmentation with Graph Convolutional Networks

Deep Neural Networks for Free-Hand Sketch Recognition.

How deep is your art: An experimental study on the limits of artistic understanding in a single-task, single-modality neural network

CLIPascene: Scene Sketching with Different Types and Levels of Abstraction

An analysis of the transfer learning of convolutional neural networks for artistic images

Open Vocabulary Semantic Scene Sketch Understanding

Abstracting Sketches through Simple Primitives