Abstract:Combining multiple observers can improve performance. This wisdom-of-the-crowd technique harnesses the independency of different observers by averaging out randomly-distributed noise. However, recent studies have shown that individuals have idiosyncratic perceptual biases, and some observers have biases that are more similar to each other. This presents the possibility that the choice of pairs when employing two observers in the same task could be optimized by measuring the perceptual biases for each observer and coupling the most independent pairs. Here we tested this hypothesis through a series of studies ranging from low-level visual localization to high-level emotion recognition tasks. In Experiment 1, observers adjusted a cursor to indicate the locations of briefly-presented noise patches. In Experiment 2, observers adjusted morphed shapes to match previously-seen shapes. In Experiment 3, observers used a cursor to dynamically track and report the emotion of a target actor in Hollywood movies. Stimulus-specific perceptual errors (the difference between perceptual report and corresponding ground truth) were measured for each observer. Emotional ground truth was defined as the consensus of an independent group of observers. We calculated the correlation between the perceptual errors for every pair of participants, within each task. To test our hypothesis, we combined the responses between each least-correlated pair of observers, using an independent set of data. This yielded significantly higher accuracies in all three studies compared to the performance of single observers or random pairs. Our results demonstrate that pairing observers according to their perceptual biases rather than sensitivity can also optimize the paired-observer performance, and this technique can be widely applied even to situations where ground truth is subjective rather than objective, such as emotion recognition. Our findings highlight the importance of measuring individual differences in perceptual biases, and they have implications for critical applications such as TSA and radiological screening.

The Wisdom of the Crowd Can Unmask Faces

Towards Mask-robust Face Recognition.

Face masks and fake masks: the effect of real and superimposed masks on face matching with super-recognisers, typical observers, and algorithms

Automated face recognition assists with low‐prevalence face identity mismatches but can bias users

Masked Face Recognition With Mask Transfer and Self-Attention Under the COVID-19 Pandemic

The effect of face masks and sunglasses on identity and expression recognition with super-recognizers and typical observers

Identity Overlap Between Face Recognition Train/Test Data: Causing Optimistic Bias in Accuracy Measurement

Computers do better than experts matching faces in a large population

DocFace+: ID Document to Selfie Matching

The Effect of Wearing a Face Mask on Face Image Quality

Trust in automation and the accuracy of human-algorithm teams performing one-to-one face matching tasks

DocFace: Matching ID Document Photos to Selfies

The Effect of Wearing a Mask on Face Recognition Performance: an Exploratory Study

Diagnostic feature training improves face matching accuracy.

A realistic approach to generate masked faces applied on two novel masked face recognition data sets

Mask-invariant Face Recognition through Template-level Knowledge Distillation

Mask-up: Investigating Biases in Face Re-identification for Masked Faces

Analysis of face detection, face landmarking, and face recognition performance with masked face images

Generating and Improving a Dataset of Masked Faces Using Data Augmentation

Masked Face Recognition Challenge: The InsightFace Track Report

Optimizing Wisdom-of-the-Crowd by Measuring Idiosyncratic Perceptual Biases