Abstract:Deliberation involves participants exchanging knowledge, arguments, and perspectives and has been shown to be effective at addressing polarization. The Stanford Online Deliberation Platform facilitates large-scale deliberations. It enables video-based online discussions on a structured agenda for small groups without requiring human moderators. This paper's data comes from various deliberation events, including one conducted in collaboration with Meta in 32 countries, and another with 38 post-secondary institutions in the US. Estimating the quality of contributions in a conversation is crucial for assessing feature and intervention impacts. Traditionally, this is done by human annotators, which is time-consuming and costly. We use a large language model (LLM) alongside eight human annotators to rate contributions based on justification, novelty, expansion of the conversation, and potential for further expansion, with scores ranging from 1 to 5. Annotators also provide brief justifications for their ratings. Using the average rating from other human annotators as the ground truth, we find the model outperforms individual human annotators. While pairs of human annotators outperform the model in rating justification and groups of three outperform it on all four metrics, the model remains competitive. We illustrate the usefulness of the automated quality rating by assessing the effect of nudges on the quality of deliberation. We first observe that individual nudges after prolonged inactivity are highly effective, increasing the likelihood of the individual requesting to speak in the next 30 seconds by 65%. Using our automated quality estimation, we show that the quality ratings for statements prompted by nudging are similar to those made without nudging, signifying that nudging leads to more ideas being generated in the conversation without losing overall quality.

Measuring the Quality of Answers in Political Q&As with Large Language Models

Measuring short-form factuality in large language models

A Quantum Expectation Value Based Language Model with Application to Question Answering

Predicting Question-Answering Performance of Large Language Models through Semantic Consistency

Leveraging Large Language Models for Multiple Choice Question Answering

Towards Measuring the Representation of Subjective Global Opinions in Language Models

Questioning the Survey Responses of Large Language Models

AQuA -- Combining Experts' and Non-Experts' Views To Assess Deliberation Quality in Online Discussions Using LLMs

PEDANTS: Cheap but Effective and Interpretable Answer Equivalence

Improving Automatic VQA Evaluation Using Large Language Models

Estimating Contribution Quality in Online Deliberations Using a Large Language Model

PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models

DelibAnalysis: Understanding the quality of online political discourse with machine learning

Leveraging Large Language Models in Code Question Answering: Baselines and Issues

A Benchmark for Long-Form Medical Question Answering

Measurement in the Age of LLMs: An Application to Ideological Scaling

Research on the Quality Prediction of Online Chinese Question Answering Community Answers Based on Comments.

Analysis of QA System Behavior against Context and Question Changes

CommunityLM: Probing Partisan Worldviews from Language Models

QuALITY: Question Answering with Long Input Texts, Yes!

Experimental Design of Extractive Question-Answering Systems: Influence of Error Scores and Answer Length