Abstract:The increasing capacities of large language models (LLMs) have been shown to present an unprecedented opportunity to scale up data analytics in the humanities and social sciences, by automating complex qualitative tasks otherwise typically carried out by human researchers. While numerous benchmarking studies have assessed the analytic prowess of LLMs, there is less focus on operationalizing this capacity for inference and hypothesis testing. Addressing this challenge, a systematic framework is argued for here, building on mixed methods quantitizing and converting design principles, and feature analysis from linguistics, to transparently integrate human expertise and machine scalability. Replicability and statistical robustness are discussed, including how to incorporate machine annotator error rates in subsequent inference. The approach is discussed and demonstrated in over a dozen LLM-assisted case studies, covering 9 diverse languages, multiple disciplines and tasks, including analysis of themes, stances, ideas, and genre compositions; linguistic and semantic annotation, interviews, text mining and event cause inference in noisy historical data, literary social network construction, metadata imputation, and multimodal visual cultural analytics. Using hypothesis-driven topic classification instead of "distant reading" is discussed. The replications among the experiments also illustrate how tasks previously requiring protracted team effort or complex computational pipelines can now be accomplished by an LLM-assisted scholar in a fraction of the time. Importantly, the approach is not intended to replace, but to augment and scale researcher expertise and analytic practices. With these opportunities in sight, qualitative skills and the ability to pose insightful questions have arguably never been more critical.

Language Models and Cognitive Automation for Economic Research

Generative AI for Economic Research: Use Cases and Implications for Economists

Large Language Models: An Applied Econometric Framework

How to Learn and Teach Economics with Large Language Models, Including GPT

Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?

Economics Arena for Large Language Models

Large Language Models for Behavioral Economics: Internal Validity and Elicitation of Mental Models

Generative LLMs and Textual Analysis in Accounting: (Chat) GPT as Research Assistant?

Towards Coding Social Science Datasets with Language Models

Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence

The economic trade-offs of large language models: A case study

Exploring the Potential of Large Language Models for Automation in Technical Customer Service

Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games

Apprentices to Research Assistants: Advancing Research with Large Language Models

LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory

Large Legislative Models: Towards Efficient AI Policymaking in Economic Simulations

Large Language Models Humanize Technology

LARGE LANGUAGE MODEL-BASED ARTIFICIAL INTELLIGENCE IN THE LANGUAGE CLASSROOM: PRACTICAL IDEAS FOR TEACHING

Potential Benefits of Employing Large Language Models in Research in Moral Education and Development

Automated Social Science: Language Models as Scientist and Subjects