Abstract:Large Language Models (LLMs) are increasingly used for boosting organizational efficiency and automating tasks. While not originally designed for complex cognitive processes, recent efforts have further extended to employ LLMs in activities such as reasoning, planning, and decision-making. In business processes, such abilities could be invaluable for leveraging on the massive corpora LLMs have been trained on for gaining deep understanding of such processes. In this work, we plant the seeds for the development of a benchmark to assess the ability of LLMs to reason about causal and process perspectives of business operations. We refer to this view as Causally-augmented Business Processes (BP^C). The core of the benchmark comprises a set of BP^C related situations, a set of questions about these situations, and a set of deductive rules employed to systematically resolve the ground truth answers to these questions. Also with the power of LLMs, the seed is then instantiated into a larger-scale set of domain-specific situations and questions. Reasoning on BP^C is of crucial importance for process interventions and process improvement. Our benchmark, accessible at <a class="link-external link-https" href="https://huggingface.co/datasets/ibm/BPC" rel="external noopener nofollow">this https URL</a>, can be used in one of two possible modalities: testing the performance of any target LLM and training an LLM to advance its capability to reason about BP^C.

A comparative study of LLMs and NLP approaches for supporting business process analysis

Conversing with business process-aware large language models: the BPLLM framework

Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis

LLM4PM: A case study on using Large Language Models for Process Modeling in Enterprise Organizations

Leveraging Large Language Models for Enhanced Process Model Comprehension

Towards a Benchmark for Large Language Models for Business Process Management Tasks

NLP4PBM: a systematic review on process extraction using natural language processing with rule-based, machine and deep learning methods

Towards a Benchmark for Causal Business Process Reasoning with LLMs

Abstractions, Scenarios, and Prompt Definitions for Process Mining with LLMs: A Case Study

A Knowledge-driven Business Process Analysis Canvas

Process Modeling With Large Language Models

Conversational Assistants in Knowledge-Intensive Contexts: An Evaluation of LLM- versus Intent-based Systems

A Reality check of the benefits of LLM in business

Comparing Generative Chatbots Based on Process Requirements

PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks

The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation

Process Modeling Leveraged by Workflow Structure and Running Logs Analysis

Making LLMs Work for Enterprise Data Tasks

Enhancing Contract Negotiations with LLM-Based Legal Document Comparison

Evaluating Large Language Models in Process Mining: Capabilities, Benchmarks, and Evaluation Strategies