Abstract:Spatial reasoning plays a vital role in both human cognition and machine intelligence, prompting new research into language models' (LMs) capabilities in this regard. However, existing benchmarks reveal shortcomings in evaluating qualitative spatial reasoning (QSR). These benchmarks typically present oversimplified scenarios or unclear natural language descriptions, hindering effective evaluation. We present a novel benchmark for assessing QSR in LMs, which is grounded in realistic 3D simulation data, offering a series of diverse room layouts with various objects and their spatial relationships. This approach provides a more detailed and context-rich narrative for spatial reasoning evaluation, diverging from traditional, toy-task-oriented scenarios. Our benchmark encompasses a broad spectrum of qualitative spatial relationships, including topological, directional, and distance relations. These are presented with different viewing points, varied granularities, and density of relation constraints to mimic real-world complexities. A key contribution is our logic-based consistency-checking tool, which enables the assessment of multiple plausible solutions, aligning with real-world scenarios where spatial relationships are often open to interpretation. Our benchmark evaluation of advanced LMs reveals their strengths and limitations in spatial reasoning. They face difficulties with multi-hop spatial reasoning and interpreting a mix of different view descriptions, pointing to areas for future improvement.

Enhancing Quantitative Reasoning Skills of Large Language Models Through Dimension Perception

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

NUMCoT: Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Models

Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data

Solving Quantitative Reasoning Problems with Language Models

Large Language Models for Mathematical Reasoning: Progresses and Challenges

CLR-Bench: Evaluating Large Language Models in College-level Reasoning

Towards Reasoning in Large Language Models: A Survey

Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models

EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference

Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems

Benchmarking Large Language Models for Math Reasoning Tasks

Reasoning in Large Language Models: A Geometric Perspective

Targeted training for numerical reasoning with large language models

FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Models

CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge

Teaching-Inspired Integrated Prompting Framework: A Novel Approach for Enhancing Reasoning in Large Language Models

Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance

Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning