Abstract:Analyzing unstructured data, such as complex documents, has been a persistent challenge in data processing. Large Language Models (LLMs) have shown promise in this regard, leading to recent proposals for declarative frameworks for LLM-powered unstructured data processing. However, these frameworks focus on reducing cost when executing user-specified operations using LLMs, rather than improving accuracy, executing most operations as-is. This is problematic for complex tasks and data, where LLM outputs for user-defined operations are often inaccurate, even with optimized prompts. We present DocETL, a system that optimizes complex document processing pipelines, while accounting for LLM shortcomings. DocETL offers a declarative interface for users to define such pipelines and uses an agent-based framework to automatically optimize them, leveraging novel agent-based rewrites (that we call {\em rewrite directives}) and an optimization and evaluation framework that we introduce. We introduce {\em (i)} logical rewriting of pipelines, tailored for LLM-based tasks, {\em (ii)} an agent-guided plan evaluation mechanism that synthesizes and orchestrates task-specific validation prompts, and {\em (iii)} an optimization algorithm that efficiently finds promising plans, considering the time constraints of LLM-based plan generation and evaluation. Our evaluation on three different unstructured document analysis tasks demonstrates that DocETL finds plans with outputs that are $1.34$ to $4.6\times$ higher quality (e.g., more accurate, comprehensive) than well-engineered baselines, addressing a critical gap in existing declarative frameworks for unstructured data analysis. DocETL is open-source at \ttt{<a class="link-external link-http" href="http://docetl.org" rel="external noopener nofollow">this http URL</a>}, and as of October 2024, has amassed over 800 GitHub Stars, with users spanning a variety of domains.

Towards Agentic Schema Refinement

Lucy: Think and Reason to Solve Text-to-SQL

Schema Matching with Large Language Models: an Experimental Study

Agent-Based Approach Towards Ontology Refinement In Virtual Enterprises

Querying Large Language Models with SQL

Compound Schema Registry

SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration

Making LLMs Work for Enterprise Data Tasks

Using off-the-shelf LLMs to query enterprise data by progressively revealing ontologies

Fine-tuning Large Enterprise Language Models via Ontological Reasoning

Exploring Autonomous Agents through the Lens of Large Language Models: A Review

Structure Guided Large Language Model for SQL Generation

Practical Considerations for Agentic LLM Systems

Here's Charlie! Realising the Semantic Web vision of Agents in the age of LLMs

Semantic Enhanced Text-to-SQL Parsing Via Iteratively Learning Schema Linking Graph

Generating Succinct Descriptions of Database Schemata for Cost-Efficient Prompting of Large Language Models

Navigating Complexity: Orchestrated Problem Solving with Multi-Agent LLMs

Solution-oriented Agent-based Models Generation with Verifier-assisted Iterative In-context Learning

Large Language Model based Multi-Agents: A Survey of Progress and Challenges

Artificial Agency and Large Language Models

DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing