Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models

Sergio E. Zanotto,Segun Aroyehun

2024-12-04

Abstract:The rapid advancements in large language models (LLMs) have significantly improved their ability to generate natural language, making texts generated by LLMs increasingly indistinguishable from human-written texts. Recent research has predominantly focused on using LLMs to classify text as either human-written or machine-generated. In our study, we adopt a different approach by profiling texts spanning four domains based on 250 distinct linguistic features. We select the M4 dataset from the Subtask B of SemEval 2024 Task 8. We automatically calculate various linguistic features with the LFTK tool and additionally measure the average syntactic depth, semantic similarity, and emotional content for each document. We then apply a two-dimensional PCA reduction to all the calculated features. Our analyses reveal significant differences between human-written texts and those generated by LLMs, particularly in the variability of these features, which we find to be considerably higher in human-written texts. This discrepancy is especially evident in text genres with less rigid linguistic style constraints. Our findings indicate that humans write texts that are less cognitively demanding, with higher semantic content, and richer emotional content compared to texts generated by LLMs. These insights underscore the need for incorporating meaningful linguistic features to enhance the understanding of textual outputs of LLMs.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to distinguish human - written texts (HWT) from texts generated by large - language models (MGT) by analyzing language features. Specifically, the researchers adopted a different method from the past. Instead of simply using large - language models for classification tasks, they conducted a detailed linguistic feature analysis of texts to find specific features that can significantly distinguish machine - generated texts from human - written texts. These features include, but are not limited to, the complexity of grammatical structures, semantic similarity, emotional content, etc. The main purpose of the research is to improve the understanding ability of text output and develop more explanatory detection tools in order to better recognize and understand the output of large - language models.

Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models

Contrasting Linguistic Patterns in Human and LLM-Generated Text

Contrasting Linguistic Patterns in Human and LLM-Generated News Text

Does a Large Language Model Really Speak in Human-Like Language?

SMLT-MUGC: Small, Medium, and Large Texts -- Machine versus User-Generated Content Detection and Comparison

Do LLMs write like humans? Variation in grammatical and rhetorical styles

Trying to be human: Linguistic traces of stochastic empathy in language models

Can Large Language Models Be an Alternative to Human Evaluations?

Inclusivity in Large Language Models: Personality Traits and Gender Bias in Scientific Abstracts

HLB: Benchmarking LLMs' Humanlikeness in Language Use

Large Language Models are Inconsistent and Biased Evaluators

Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas

Generative AI Text Classification using Ensemble LLM Approaches

Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models

Benchmarking Linguistic Diversity of Large Language Models

Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function

The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?

From Text to Source: Results in Detecting Large Language Model-Generated Content

Style Over Substance: Evaluation Biases for Large Language Models

Under the Surface: Tracking the Artifactuality of LLM-Generated Data