Abstract:Deep generative models such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Diffusion Models, and Transformers, have shown great promise in a variety of applications, including image and speech synthesis, natural language processing, and drug discovery. However, when applied to engineering design problems, evaluating the performance of these models can be challenging, as traditional statistical metrics based on likelihood may not fully capture the requirements of engineering applications. This paper doubles as a review and practical guide to evaluation metrics for deep generative models (DGMs) in engineering design. We first summarize the well-accepted `classic' evaluation metrics for deep generative models grounded in machine learning theory. Using case studies, we then highlight why these metrics seldom translate well to design problems but see frequent use due to the lack of established alternatives. Next, we curate a set of design-specific metrics which have been proposed across different research communities and can be used for evaluating deep generative models. These metrics focus on unique requirements in design and engineering, such as constraint satisfaction, functional performance, novelty, and conditioning. Throughout our discussion, we apply the metrics to models trained on simple-to-visualize 2-dimensional example problems. Finally, we evaluate four deep generative models on a bicycle frame design problem and structural topology generation problem. In particular, we showcase the use of proposed metrics to quantify performance target achievement, design novelty, and geometric constraints. We publicly release the code for the datasets, models, and metrics used throughout the paper at <a class="link-external link-https" href="https://decode.mit.edu/projects/metrics/" rel="external noopener nofollow">this https URL</a>.

A Unifying Information-theoretic Perspective on Evaluating Generative Models

A Study on the Evaluation of Generative Models

Unifying and extending Precision Recall metrics for assessing generative models

Attribute Based Interpretable Evaluation Metrics for Generative Models

Analyzing Generative Models by Manifold Entropic Metrics

How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

An Interpretable Evaluation of Entropy-based Novelty of Generative Models

Emergent Asymmetry of Precision and Recall for Measuring Fidelity and Diversity of Generative Models in High Dimensions

An empirical study on evaluation metrics of generative adversarial networks

Rethinking the Evaluation of Unbiased Scene Graph Generation

Towards Robust Evaluation of Protein Generative Models: A Systematic Analysis of Metrics

Measuring Orthogonality in Representations of Generative Models

We Need to Talk About Classification Evaluation Metrics in NLP

Measuring Diversity in Co-creative Image Generation

A Distributional Evaluation of Generative Image Models

Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design

Towards a Scalable Reference-Free Evaluation of Generative Models

On the Evaluation of Generative Adversarial Networks By Discriminative Models

A domain agnostic measure for monitoring and evaluating GANs

Image Generation Diversity Issues and How to Tame Them

Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation