Abstract:The recent popularity of large language models (LLMs) has brought a significant impact to boundless fields, particularly through their open-ended ecosystem such as the APIs, open-sourced models, and plugins. However, with their widespread deployment, there is a general lack of research that thoroughly discusses and analyzes the potential risks concealed. In that case, we intend to conduct a preliminary but pioneering study covering the robustness, consistency, and credibility of LLMs systems. With most of the related literature in the era of LLM uncharted, we propose an automated workflow that copes with an upscaled number of queries/responses. Overall, we conduct over a million queries to the mainstream LLMs including ChatGPT, LLaMA, and OPT. Core to our workflow consists of a data primitive, followed by an automated interpreter that evaluates these LLMs under different adversarial metrical systems. As a result, we draw several, and perhaps unfortunate, conclusions that are quite uncommon from this trendy community. Briefly, they are: (i)-the minor but inevitable error occurrence in the user-generated query input may, by chance, cause the LLM to respond unexpectedly; (ii)-LLMs possess poor consistency when processing semantically similar query input. In addition, as a side finding, we find that ChatGPT is still capable to yield the correct answer even when the input is polluted at an extreme level. While this phenomenon demonstrates the powerful memorization of the LLMs, it raises serious concerns about using such data for LLM-involved evaluation in academic development. To deal with it, we propose a novel index associated with a dataset that roughly decides the feasibility of using such data for LLM-involved evaluation. Extensive empirical studies are tagged to support the aforementioned claims.

A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attribution

Can Large Language Models Identify Authorship?

Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges

Neural Authorship Attribution: Stylometric Analysis on Large Language Models

Sui Generis: Large Language Models for Authorship Attribution and Verification in Latin

ALMs: Authorial Language Models for Authorship Attribution

LLM Attributor: Interactive Visual Attribution for LLM Generation

Authorship attribution based on a probabilistic topic model

Enhancing Authorship Attribution through Embedding Fusion: A Novel Approach with Masked and Encoder-Decoder Language Models

Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference

Evaluation of Attribution Bias in Retrieval-Augmented Large Language Models

AIDBench: A benchmark for evaluating the authorship identification capability of large language models

Integrating Bidirectional Long Short-Term Memory with Subword Embedding for Authorship Attribution

Bayesian Statistical Modeling with Predictors from LLMs

Bayesian Reward Models for LLM Alignment

Attribute or Abstain: Large Language Models as Long Document Assistants

Latent Space Interpretation for Stylistic Analysis and Explainable Authorship Attribution

T5 meets Tybalt: Author Attribution in Early Modern English Drama Using Large Language Models

Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility

Source Attribution for Large Language Model-Generated Data

InstructAV: Instruction Fine-tuning Large Language Models for Authorship Verification