Abstract:Logical reasoning is a fundamental task in natural language processing that presents significant challenges to Large Language Models (LLMs). The inherent characteristics of logical reasoning makes it well-suited for symbolic representations such as first-order logic (FOL). Research in symbolic logical reasoning explored FOL generation using state-of-the-art LLMs (i.e., GPT-4) to produce FOL translations of natural language (NL) statements, but errors in translation are usually not the focus. We address this by categorizing the translation errors in FOL statements generated by LLMs. To make progress towards improving the quality of FOL translations for smaller language models such as LLaMA-2 13B and Mistral 7B, we create ProofFOL, a high-quality FOL-annotated subset of ProofWriter dataset using GPT-4o. The models fine-tuned on this silver standard data achieve a significant gain in performance when compared to larger language models such as LLaMA-2 70B. In addition to improving the model using large data, we also tackle the issue of data scarcity and introduce an incremental framework encompassing of data augmentation and verification steps. In the augmentation process, a single pair of (premises, conclusion) is split into multiple new instances based on the predicates and FOLs. This data is used for fine-tuning, and the inference on this model generates FOLs with fewer errors over the model trained on the original data. Our investigation on the translation errors leads to generation of a perturbation dataset, which is used to train a verifier that corrects potential syntactic and semantic FOL translation errors. We demonstrate an efficient method for making the most of a limited existing human-annotated dataset. Our results show state-of-the-art performance for ProofWriter and ProntoQA datasets using ProofFOL on LLaMA-2 and Mistral models.

Topologically sorting VDM-SL definitions for Isabelle/HOL translation

VDM recursive functions in Isabelle/HOL

Formal Foundations for Translational Separation Logic Verifiers (extended version)

The VMT-LIB Language and Tools

A Framework for Bottom-Up Simulation of SLD-Resolution

Translating HOL to Dedukti

Specification-based CSV Support in VDM

Experiences from Exporting Major Proof Assistant Libraries

Isabelle/HOL as a Meta-Language for Teaching Logic

Implementation-First Approach of Developing Formal Semantics of a Simulation Language in VDM-SL

Scalable Automated Verification for Cyber-Physical Systems in Isabelle/HOL

Mechanization of LAGC Semantics in Isabelle

Translating SUMO-K to Higher-Order Set Theory

Canonical translation surfaces for computing Veech groups

Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verification

Translating Hierarchical Block Diagrams into Composite Predicate Transformers

Topological Semantics for Lumped Parameter Systems Modeling

Infinite Words and Morphic Languages Formalized in Isabelle/HOL

An overview of Generalised Veltman Semantics

Sharing proofs with predicative theories through universe-polymorphic elaboration

Type Theory as a Language Workbench