Abstract:Analyzing unstructured data, such as complex documents, has been a persistent challenge in data processing. Large Language Models (LLMs) have shown promise in this regard, leading to recent proposals for declarative frameworks for LLM-powered unstructured data processing. However, these frameworks focus on reducing cost when executing user-specified operations using LLMs, rather than improving accuracy, executing most operations as-is. This is problematic for complex tasks and data, where LLM outputs for user-defined operations are often inaccurate, even with optimized prompts. We present DocETL, a system that optimizes complex document processing pipelines, while accounting for LLM shortcomings. DocETL offers a declarative interface for users to define such pipelines and uses an agent-based framework to automatically optimize them, leveraging novel agent-based rewrites (that we call {\em rewrite directives}) and an optimization and evaluation framework that we introduce. We introduce {\em (i)} logical rewriting of pipelines, tailored for LLM-based tasks, {\em (ii)} an agent-guided plan evaluation mechanism that synthesizes and orchestrates task-specific validation prompts, and {\em (iii)} an optimization algorithm that efficiently finds promising plans, considering the time constraints of LLM-based plan generation and evaluation. Our evaluation on three different unstructured document analysis tasks demonstrates that DocETL finds plans with outputs that are $1.34$ to $4.6\times$ higher quality (e.g., more accurate, comprehensive) than well-engineered baselines, addressing a critical gap in existing declarative frameworks for unstructured data analysis. DocETL is open-source at \ttt{<a class="link-external link-http" href="http://docetl.org" rel="external noopener nofollow">this http URL</a>}, and as of October 2024, has amassed over 800 GitHub Stars, with users spanning a variety of domains.

Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes

Evalverse: Unified and Accessible Library for Large Language Model Evaluation

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows

Are Large Language Models the New Interface for Data Pipelines?

Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges

JarviX: A LLM No code Platform for Tabular Data Analysis and Optimization

A Virtual Dataspaces Model for Large-Scale Materials Scientific Data Access

Scaling Up Video Summarization Pretraining with Large Language Models

DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articles

METL: a modern ETL pipeline with a dynamic mapping matrix

Automatic Data Transformation Using Large Language Model: An Experimental Study on Building Energy Data

Towards Enabling FAIR Dataspaces Using Large Language Models

DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models

LinkTransformer: A Unified Package for Record Linkage with Transformer Language Models

Harnessing Scalable Transactional Stream Processing for Managing Large Language Models [Vision]

DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing

DOLLmC: DevOps for Large Language model Customization

Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Models

UniDM: A Unified Framework for Data Manipulation with Large Language Models

VideoLLM: Modeling Video Sequence with Large Language Models