A Statistical Turing Test for Generative Models

Hayden Helm,Carey E. Priebe,Weiwei Yang

DOI: https://doi.org/10.48550/arXiv.2309.08913

2023-09-16

Abstract:The emergence of human-like abilities of AI systems for content generation in domains such as text, audio, and vision has prompted the development of classifiers to determine whether content originated from a human or a machine. Implicit in these efforts is an assumption that the generation properties of a human are different from that of the machine. In this work, we provide a framework in the language of statistical pattern recognition that quantifies the difference between the distributions of human and machine-generated content conditioned on an evaluation context. We describe current methods in the context of the framework and demonstrate how to use the framework to evaluate the progression of generative models towards human-like capabilities, among many axes of analysis.

Artificial Intelligence,Computation and Language,Computers and Society,Machine Learning

What problem does this paper attempt to address?

The core problem that this paper attempts to solve is to quantify the differences between machine - generated content and human - generated content, and provide a statistical framework to evaluate such differences. Specifically, the paper proposes a framework of Statistical Turing Test, which is used to measure the degree to which different generation models can imitate human abilities in specific evaluation contexts. This framework can not only evaluate the performance of a single model, but also compare the progress among different model families, as well as the effectiveness of different classifiers in human - detection tasks.

### Main Contributions
1. **Statistical Framework**: The paper proposes a statistical framework for quantifying the detectability of machines in a given human - detection context. This framework does not depend on content modalities and generation tasks, and provides a general language to analyze important aspects of the human - detection problem, such as the progress of model families towards human abilities and the effectiveness of different classifiers.
2. **Human - Detection Method**: Besides the framework itself, the paper also contributes a new zero - sample human - detection method - ProxiHuman. This method utilizes the geometric properties of the machine - embedding space to determine the source of content (human or machine).

### Framework Overview
- **Problem Definition**: Suppose there is a set of content - label pairs \((x_1,y_1),\ldots,(x_n,y_n)\), where the label \(y_i\in\{0, 1\}\) indicates whether the content \(x_i\) is generated by a machine (label 0) or a human (label 1). The goal of the human - detection problem is to construct a classifier \(h:X\rightarrow\{0, 1\}\) to correctly judge whether the content is generated by a human or a machine.
- **Risk Function**: The risk \(R(h;P,\ell)\) of the classifier \(h\) is defined as the expected loss \(E_P[\ell(h(X),Y)]\), where \(P\) is the classification distribution and \(\ell\) is the loss function.
- **Optimal Classifier**: Define \(h^*\) as the classifier with the minimum risk in the set \(H\), and \(h^{**}\) as the classifier with the minimum risk among all functions from \(X\) to \(\{0, 1\}\).

### Human - Detection Context
- **Definition**: The human - detection context \(C\) is a six - tuple \((X,f_1,\pi,\ell,t,H)\), where:
- \(X\) is the sample space.
- \(f_1\) is the class - conditional distribution of human - generated content.
- \(\pi\) is the class - conditional prior.
- \(\ell\) is the loss function.
- \(t\) is the transformation of the input space.
- \(H\) is the set of classifiers.
- **Undetectability**: The machine - parameterized content - generation distribution \(f_0\) is \(\tau\)-undetectable in the context \(C\) if \(\frac{1 - R(h^*_t;P,\ell)}{R(h_c;P,\ell)}\leq\tau\), where \(h_c\) is the random - guessing classifier.

### Experimental Evaluation
- **Dataset**: The paper uses four natural - language datasets: GPT - wiki - intro, XSum, WritingPrompts and PubMed QA.
- **Detection Method**: Four different detection methods are evaluated: Likelihood, LogRank, DetectGPT and ProxiHuman.
- **Result**: The experimental results show that the content generated by GPT4 in all four contexts is closer to human - generated content than that generated by GPT3. For each transformation method, the content generated by GPT4 is more difficult to distinguish from human content.

Through these contributions, the paper provides a systematic framework for evaluating and comparing the human - detection abilities of generation models, which is helpful for understanding the performance of these models in different tasks and contexts.