Abstract:Temporal logics are powerful tools that are widely used for the synthesis and verification of reactive systems. The recent progress on Large Language Models (LLMs) has the potential to make the process of writing such specifications more accessible. However, writing specifications in temporal logics remains challenging for all but the most expert users. A key question in using LLMs for temporal logic specification engineering is to understand what kind of guidance is most helpful to the LLM and the users to easily produce specifications. Looking specifically at the problem of reactive program synthesis, we explore the impact of providing an LLM with guidance on the separation of control and data--making explicit for the LLM what functionality is relevant for the specification, and treating the remaining functionality as an implementation detail for a series of pre-defined functions and predicates. We present a benchmark set and find that this separation of concerns improves specification generation. Our benchmark provides a test set against which to verify future work in LLM generation of temporal logic specifications.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use large - language models (LLMs) to simplify and improve the writing process of temporal logic (especially temporal stream logic, TSL) specifications. Specifically, the paper focuses on the following aspects: 1. **Reducing the writing difficulty**: Traditional temporal logic specification writing is very difficult for non - expert users, and LLMs have the potential to make this process easier and more intuitive. 2. **Separating control and data**: By clearly distinguishing control logic and data processing, the paper explores how to guide LLMs to generate more accurate TSL specifications. This includes defining function and predicate terms and abstracting out irrelevant implementation details. 3. **Improving the accuracy of specification generation**: The paper proposes a new method to help LLMs better understand and generate correct TSL specifications by providing natural - language descriptions and predefined functional interfaces. ### Specific problems - **How can LLMs be made to better understand and generate temporal logic specifications?** - The paper guides LLMs to generate TSL specifications by providing natural - language summaries, detailed descriptions, and definitions of functions and predicates. - **Can separating control logic and data processing improve the accuracy of LLMs in generating specifications?** - Research shows that in some cases, this separation does improve the accuracy of generated specifications, especially when LLMs can correctly understand the definitions of functions and predicates. - **How to evaluate the effectiveness of this method?** - The paper designs a series of benchmark tests to evaluate the impact of different information inputs on specification generation and measures the quality of generated specifications by multiple indicators. ### Main contributions 1. **Proposing a TSL specification - generation pipeline based on LLMs**, which guides specification generation through natural - language descriptions and predefined functional interfaces. 2. **Creating a set of benchmark tests** to verify the effectiveness and practicality of this pipeline. 3. **Observing that separating control logic and data processing helps to improve the accuracy of LLMs in generating specifications**, but only if LLMs correctly understand these definitions. ### Conclusion The paper experimentally verifies that the method of separating control logic and data processing can significantly improve the accuracy of LLMs in generating TSL specifications in some cases. However, this also depends on the ability of LLMs to understand the definitions of functions and predicates. Future work can further explore how to optimize the understanding of these definitions by LLMs, thereby further improving the quality of specification generation.

Guiding LLM Temporal Logic Generation with Explicit Separation of Data and Control

Combining LLM Code Generation with Formal Specifications and Reactive Program Synthesis

Formally Specifying the High-Level Behavior of LLM-Based Agents

Towards Neural-Guided Program Synthesis for Linear Temporal Logic Specifications

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

Specifications: The missing link to making the development of LLM systems an engineering discipline

SpecLLM: Exploring Generation and Review of VLSI Design Specification with Large Language Model

Robust, Expressive, and Quantitative Linear Temporal Logics: Pick any Two for Free

Robust, Expressive, and Quantitative Linear Temporal Logics: Pick any Two for Free (full version)

Temporal Stream Logic: Synthesis Beyond the Bools

Taming Large Bounds in Synthesis from Bounded-Liveness Specifications (Full Version)

Standpoint Linear Temporal Logic

Formal-LLM: Integrating Formal Language and Natural Language for Controllable LLM-based Agents

Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guarantees

Specification sketching for Linear Temporal Logic

Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning

Reactive Synthesis Modulo Theories Using Abstraction Refinement

Back to the Future: Towards Explainable Temporal Reasoning with Large Language Models

Low-code LLM: Graphical User Interface over Large Language Models

Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environments

LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks