Abstract:Recent advancements in large language models (LLMs) have driven a revolutionary paradigm shift in process automation from Robotic Process Automation to Agentic Process Automation by automating the workflow orchestration procedure based on LLMs. However, existing LLMs (even the advanced OpenAI GPT-4o) are confined to achieving satisfactory capability in workflow orchestration. To address this limitation, we present WorkflowLLM, a data-centric framework elaborately designed to enhance the capability of LLMs in workflow orchestration. It first constructs a large-scale fine-tuning dataset WorkflowBench with 106,763 samples, covering 1,503 APIs from 83 applications across 28 categories. Specifically, the construction process can be divided into three phases: (1) Data Collection: we collect real-world workflow data from Apple Shortcuts and RoutineHub, transcribing them into Python-style code. We further equip them with generated hierarchical thought via ChatGPT. (2) Query Expansion: we prompt ChatGPT to generate more task queries to enrich the diversity and complexity of workflows. (3) Workflow Generation: we leverage an annotator model trained on collected data to generate workflows for synthesized queries. Finally, we merge the synthetic samples that pass quality confirmation with the collected samples to obtain the WorkflowBench. Based on WorkflowBench, we fine-tune Llama-3.1-8B to obtain WorkflowLlama. Our experiments show that WorkflowLlama demonstrates a strong capacity to orchestrate complex workflows, while also achieving notable generalization performance on previously unseen APIs. Additionally, WorkflowBench exhibits robust zero-shot generalization capabilities on an out-of-distribution task planning dataset, T-Eval. Our data and code are available at <a class="link-external link-https" href="https://github.com/OpenBMB/WorkflowLLM" rel="external noopener nofollow">this https URL</a>.

LLM4Workflow: An LLM-based Automated Workflow Model Generation Tool

WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models

AutoFlow: Automated Workflow Generation for Large Language Model Agents

LLMs for science: Usage for code generation and data analysis

Low-code LLM: Graphical User Interface over Large Language Models

FlowMind: Automatic Workflow Generation with LLMs

Benchmarking Agentic Workflow Generation

LLM-for-X: Application-agnostic Integration of Large Language Models to Support Personal Writing Workflows

AFlow: Automating Agentic Workflow Generation

Agent Aided Workflow Modeling

A Survey on LLM-based Multi-Agent Systems: Workflow, Infrastructure, and Challenges

OverleafCopilot: Empowering Academic Writing in Overleaf with Large Language Models

Requirements are All You Need: From Requirements to Code with LLMs

Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey

S3LLM: Large-Scale Scientific Software Understanding with LLMs using Source, Metadata, and Document

An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project

On the effectiveness of Large Language Models for GitHub Workflows

Planning-Driven Programming: A Large Language Model Programming Workflow

Combining LLM Code Generation with Formal Specifications and Reactive Program Synthesis

Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents

LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systems