Abstract:Large Language Models (LLMs) have demonstrated remarkable capabilities across textual and visual domains but often generate outputs that violate physical laws, revealing a gap in their understanding of the physical world. Inspired by human cognition, where perception is fundamental to reasoning, we explore augmenting LLMs with enhanced perception abilities using Internet of Things (IoT) sensor data and pertinent knowledge for IoT task reasoning in the physical world. In this work, we systematically study LLMs capability to address real-world IoT tasks by augmenting their perception and knowledge base, and then propose a unified framework, IoT-LLM, to enhance such capability. In IoT-LLM, we customize three steps for LLMs: preprocessing IoT data into formats amenable to LLMs, activating their commonsense knowledge through chain-of-thought prompting and specialized role definitions, and expanding their understanding via IoT-oriented retrieval-augmented generation based on in-context learning. To evaluate the performance, We design a new benchmark with five real-world IoT tasks with different data types and reasoning difficulties and provide the benchmarking results on six open-source and close-source LLMs. Experimental results demonstrate the limitations of existing LLMs with naive textual inputs that cannot perform these tasks effectively. We show that IoT-LLM significantly enhances the performance of IoT tasks reasoning of LLM, such as GPT-4, achieving an average improvement of 65% across various tasks against previous methods. The results also showcase LLMs ability to comprehend IoT data and the physical law behind data by providing a reasoning process. Limitations of our work are claimed to inspire future research in this new era.

Parallel Collaborative Reasoning Approaches Based on DatalogMTL in IoT Scenarios

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

Goal-Driven Reasoning in DatalogMTL with Magic Sets

Practical Reasoning in DatalogMTL

Semantic Reasoning for Context-Aware Internet of Things Applications

IoT-LLM: Enhancing Real-World IoT Task Reasoning with Large Language Models

Enhancing Human-AI Collaboration Through Logic-Guided Reasoning

Scalable Distributed Ontology Reasoning Using DHT-Based Partitioning

The Datalog(Dl) Combination Of Deduction Rules And Description Logics

When Do Program-of-Thought Works for Reasoning?

When IoT Meet LLMs: Applications and Challenges

A Multi-View Temporal Knowledge Graph Reasoning Framework with Interpretable Logic Rules and Feature Fusion

Semantical Markov Logic Network for Distributed Reasoning in Cyber-Physical Systems.

Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning

Distributed Reasoning with Fuzzy Description Logics

Knowledge-based Reasoning and Learning under Partial Observability in Ad Hoc Teamwork

Exploiting Ontological Reasoning In Argumentation Based Multi-Agent Collaborative Classification

A graph-based spatial temporal logic for knowledge representation and automated reasoning in cognitive robots

Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoning

DatalogDL: Datalog Rules Parameterized by Description Logics