Abstract:Perceptual metrics, like the Fréchet Inception Distance (FID), are widely used to assess the similarity between synthetically generated and ground truth (real) images. The key idea behind these metrics is to compute errors in a deep feature space that captures perceptually and semantically rich image features. Despite their popularity, the effect that different deep features and their design choices have on a perceptual metric has not been well studied. In this work, we perform a causal analysis linking differences in semantic attributes and distortions between face image distributions to Fréchet distances (FD) using several popular deep feature spaces. A key component of our analysis is the creation of synthetic counterfactual faces using deep face generators. Our experiments show that the FD is heavily influenced by its feature space's training dataset and objective function. For example, FD using features extracted from ImageNet-trained models heavily emphasize hats over regions like the eyes and mouth. Moreover, FD using features from a face gender classifier emphasize hair length more than distances in an identity (recognition) feature space. Finally, we evaluate several popular face generation models across feature spaces and find that StyleGAN2 consistently ranks higher than other face generators, except with respect to identity (recognition) features. This suggests the need for considering multiple feature spaces when evaluating generative models and using feature spaces that are tuned to nuances of the domain of interest.

Fréchet Wavelet Distance: A Domain-Agnostic Metric for Image Generation

Frechet Music Distance: A Metric For Generative Symbolic Music Evaluation

Rethinking FID: Towards a Better Evaluation Metric for Image Generation

FLD+: Data-efficient Evaluation Metric for Generative Models

Compound Frechet Inception Distance for Quality Assessment of GAN Created Images

Fréchet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos

Wavelet-Packets for Deepfake Image Analysis and Detection

Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality

Evaluating generative networks using Gaussian mixtures of image features

Frećhet Denoised Distance: Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencoder

FFAD: A Novel Metric for Assessing Generated Time Series Data Utilizing Fourier Transform and Auto-encoder

F?D: On understanding the role of deep feature spaces on face generation evaluation

Using Skew to Assess the Quality of GAN-generated Image Features

Adapting Frechet Audio Distance for Generative Music Evaluation

The Role of ImageNet Classes in Fréchet Inception Distance

Efficient image generation with Contour Wavelet Diffusion

On the Content Bias in Fréchet Video Distance

Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation

Normalizing Flow-Based Metric for Image Generation