Abstract:Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. To investigate this, we propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic, comprising both primitive and compositional rules across five domains. Our analysis of GPT-series models over a rule subset reveals significant gaps in LLMs' logic understanding compared to human performance, especially in compositional and structural complex rules with certain bias patterns. We further distill these rules into a smaller-scale inference engine for flexible rule generation and enhancing downstream reasoning. Through a multi-judger evaluation, our inference engine proves effective in generating accurate, complex and abstract conclusions and premises, and improve various commonsense reasoning tasks. Overall, our work sheds light on LLMs' limitations in grasping inferential rule and suggests ways to enhance their logical reasoning abilities~\footnote{Code and data are available at \url{<a class="link-external link-https" href="https://github.com/SiyuanWangw/ULogic" rel="external noopener nofollow">this https URL</a>}.}.

An Extension To Owl With General Rules

The expressive language ALCNHR+K(D) for knowledge reasoning

The Semantic Web Rule Language Expressiveness Extensions-A Survey

If-Then and If-Then-Unless Rules in the Semantic Web

Translating OWL and Semantic Web Rules into Prolog: Moving Toward Description Logic Programs

Discovering Anomalies in Semantic Web Rules

The Extensions of OWL on Properties Exclusion and the Corresponding Semantic Parser

Logical foundation of semantic web

Reasoning Makes Good Annotators : an Automatic Task-specific Rules Distilling Framework for Low-resource Relation Extraction

The Datalog(Dl) Combination Of Deduction Rules And Description Logics

Founded Semantics and Constraint Semantics of Logic Rules

Alc(P)(U): An Integration Of Description Logic And General Rules

A Fuzzy Extension of OWL Based on F-ALC(G)

Ontology as a Source for Rule Generation

A fuzzy extension of the SROIQ(G) DL

Integrating Logic Rules with Everything Else, Seamlessly

Ontology-based Axiomatic System Design Semantic Modeling and Reasoning Rules

Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs

DatalogDL: Datalog Rules Parameterized by Description Logics

A Rule-based Approach for Maintenance of OWL-S Ontology Syntactic Consistency

Implementing and Evaluating a Rule-Based Approach to Querying Regular EL+ Ontologies