Seeing Faces in Things: A Model and Dataset for Pareidolia

Mark Hamilton,Simon Stent,Vasha DuTell,Anne Harrington,Jennifer Corbett,Ruth Rosenholtz,William T. Freeman

2024-09-24

Abstract:The human visual system is well-tuned to detect faces of all shapes and sizes. While this brings obvious survival advantages, such as a better chance of spotting unknown predators in the bush, it also leads to spurious face detections. ``Face pareidolia'' describes the perception of face-like structure among otherwise random stimuli: seeing faces in coffee stains or clouds in the sky. In this paper, we study face pareidolia from a computer vision perspective. We present an image dataset of ``Faces in Things'', consisting of five thousand web images with human-annotated pareidolic faces. Using this dataset, we examine the extent to which a state-of-the-art human face detector exhibits pareidolia, and find a significant behavioral gap between humans and machines. We find that the evolutionary need for humans to detect animal faces, as well as human faces, may explain some of this gap. Finally, we propose a simple statistical model of pareidolia in images. Through studies on human subjects and our pareidolic face detectors we confirm a key prediction of our model regarding what image conditions are most likely to induce pareidolia. Dataset and Website: <a class="link-external link-https" href="https://aka.ms/faces-in-things" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Artificial Intelligence,Human-Computer Interaction,Information Retrieval,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to understand and model the "face pareidolia" phenomenon in the human visual system from the perspective of computer vision, and explore the manifestation of this phenomenon in machine - learning models and improvement methods**.

Specifically, the paper focuses on the following aspects:

1. **Defining and understanding Pareidolia**:
- Face pareidolia refers to the phenomenon of perceiving face - like structures in random or meaningless stimuli, for example, seeing a face in a coffee stain or a cloud.
- The author points out that although this phenomenon is common in humans and has certain evolutionary advantages (such as quickly identifying potential threats), it has not been fully studied in computer vision systems.

2. **Constructing a dataset**:
- The paper introduces a dataset named "Faces in Things", which contains 5,000 images labeled with face pareidolia. These images are collected from the Internet and manually labeled for the study of face pareidolia.

3. **Evaluating the performance of existing models**:
- Using the "Faces in Things" dataset, the author evaluates the detection ability of the state - of - the - art face detection models (such as RetinaFace) for face pareidolia. The results show that these models perform poorly in detecting pareidolia, and even relaxing the detection threshold does not significantly improve the situation.

4. **Exploring methods to improve models**:
- The author improves the detection ability of existing face detection models for face pareidolia by fine - tuning them (training with animal face data). The experimental results show that adding animal face data can significantly improve the performance of the model, which may be related to the broad detection needs for multiple faces in human evolution.

5. **Proposing mathematical models of pareidolia**:
- The paper proposes two simple mathematical models to describe the occurrence mechanism of face pareidolia. One is a model based on the Gaussian process, and the other is a model based on deep features. Both models predict the existence of an "optimal interval", within which the image conditions are most likely to induce face pareidolia.

6. **Verifying model predictions**:
- Through human subject experiments and computer vision model tests, the author verifies the key predictions of the proposed models regarding which image conditions are most likely to induce face pareidolia.

In summary, this paper aims to fill the gap in the research on face pareidolia in the field of computer vision, providing a new dataset and theoretical framework to help understand and improve the machine's detection ability of face pareidolia.