Integrating Explanations in Learning LTL Specifications from Demonstrations

Ashutosh Gupta,John Komp,Abhay Singh Rajput,Krishna Shankaranarayanan,Ashutosh Trivedi,Namrita Varshney

2024-04-04

Abstract:This paper investigates whether recent advances in Large Language Models (LLMs) can assist in translating human explanations into a format that can robustly support learning Linear Temporal Logic (LTL) from demonstrations. Both LLMs and optimization-based methods can extract LTL specifications from demonstrations; however, they have distinct limitations. LLMs can quickly generate solutions and incorporate human explanations, but their lack of consistency and reliability hampers their applicability in safety-critical domains. On the other hand, optimization-based methods do provide formal guarantees but cannot process natural language explanations and face scalability challenges. We present a principled approach to combining LLMs and optimization-based methods to faithfully translate human explanations and demonstrations into LTL specifications. We have implemented a tool called Janaka based on our approach. Our experiments demonstrate the effectiveness of combining explanations with demonstrations in learning LTL specifications through several case studies.

Artificial Intelligence

What problem does this paper attempt to address?

This paper discusses the integration of large language models (LLMs) with optimization methods to more effectively learn linear temporal logic (LTL) specifications from demonstrations and explanations. Existing methods mainly rely on optimization-based search to extract LTL specifications, but they may fail to address natural language interpretation and scalability issues. The paper proposes an approach that utilizes LLMs to generate possible LTL formulas and evaluates and improves the adaptability of these formulas through quantized semantics. The experiments demonstrate that the combination of explanation and demonstration can improve the accuracy and efficiency of LTL specification learning.

Integrating Explanations in Learning LTL Specifications from Demonstrations

Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement

From large language models to small logic programs: building global explanations from disagreeing local post-hoc explainers

From Feature Importance to Natural Language Explanations Using LLMs with RAG

In-Context Explainers: Harnessing LLMs for Explaining Black Box Models

Towards Uncovering How Large Language Model Works: An Explainability Perspective

Evaluating Explanations Through LLMs: Beyond Traditional User Studies

Formal Specifications from Natural Language

Complementary Explanations for Effective In-Context Learning

LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations

DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications

LLM-Generated Black-box Explanations Can Be Adversarially Helpful

Constrained LTL Specification Learning from Examples

Training LLMs to Better Self-Debug and Explain Code

A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters

Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environments

Accelerating human-computer interaction through convergent conditions for LLM explanation

FaithLM: Towards Faithful Explanations for Large Language Models

Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach