Tilt your Head: Activating the Hidden Spatial-Invariance of Classifiers

Johann Schmidt,Sebastian Stober

2024-05-06

Abstract:Deep neural networks are applied in more and more areas of everyday life. However, they still lack essential abilities, such as robustly dealing with spatially transformed input signals. Approaches to mitigate this severe robustness issue are limited to two pathways: Either models are implicitly regularised by increased sample variability (data augmentation) or explicitly constrained by hard-coded inductive biases. The limiting factor of the former is the size of the data space, which renders sufficient sample coverage intractable. The latter is limited by the engineering effort required to develop such inductive biases for every possible scenario. Instead, we take inspiration from human behaviour, where percepts are modified by mental or physical actions during inference. We propose a novel technique to emulate such an inference process for neural nets. This is achieved by traversing a sparsified inverse transformation tree during inference using parallel energy-based evaluations. Our proposed inference algorithm, called Inverse Transformation Search (ITS), is model-agnostic and equips the model with zero-shot pseudo-invariance to spatially transformed inputs. We evaluated our method on several benchmark datasets, including a synthesised ImageNet test set. ITS outperforms the utilised baselines on all zero-shot test scenarios.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the robustness issue of deep neural networks when processing spatially - transformed input signals. Specifically, when the input data undergoes spatial transformations such as rotation, scaling, and shearing, existing deep - learning models often fail to maintain the consistency and accuracy of predictions. This is mainly because current methods either rely on data augmentation to increase the diversity of samples or introduce inductive biases through hard - coding, but these methods all have their own limitations. For example, data augmentation is limited by the size of the data space and it is difficult to cover all possible changes; while hard - coded inductive biases require a large amount of engineering design for each scenario.

To solve this problem, the paper proposes a new inference technique - Inverse Transformation Search (ITS). This technique simulates human behavior when recognizing objects, that is, adjusting the perceived image through actions such as "tilting the head" or "taking a step forward", thereby achieving pseudo - invariance to the spatial transformation of the input signal. Specifically, the ITS algorithm traverses a sparsified inverse - transformation tree during the inference process and uses parallel energy evaluations to select the most appropriate transformation, enabling the model to still maintain good prediction performance under unseen spatial transformations.

The main contributions of the paper are:
1. **Zero - shot pseudo - invariance**: The ITS algorithm can make the model pseudo - invariant to spatial transformations without additional training.
2. **Model - agnosticism**: The ITS algorithm is applicable to any pre - trained classifier, only requiring access to its logit scores.
3. **Efficiency**: Through techniques such as curvature estimation of local maxima and Monte Carlo dropout, the ITS algorithm can effectively reduce noise and improve the stability of confidence - estimate.

The paper conducted experiments on multiple benchmark datasets, including the synthetic ImageNet test set, and the results show that the ITS algorithm outperforms the baseline methods in all zero - shot test scenarios.