Abstract:Mathematical reasoning remains an ongoing challenge for AI models, especially for geometry problems that require both linguistic and visual signals. As the vision encoders of most MLLMs are trained on natural scenes, they often struggle to understand geometric diagrams, performing no better in geometry problem solving than LLMs that only process text. This limitation is amplified by the lack of effective methods for representing geometric relationships. To address these issues, we introduce the Diagram Formalization Enhanced Geometry Problem Solver (DFE-GPS), a new framework that integrates visual features, geometric formal language, and natural language representations. We propose a novel synthetic data approach and create a large-scale geometric dataset, SynthGeo228K, annotated with both formal and natural language captions, designed to enhance the vision encoder for a better understanding of geometric structures. Our framework improves MLLMs' ability to process geometric diagrams and extends their application to open-ended tasks on the formalgeo7k dataset.

What problem does this paper attempt to address?

The problem this paper attempts to address is: how to improve the performance of multimodal large language models (MLLMs) in solving geometric problems, especially when dealing with geometric figures. Specifically, the paper points out that current MLLMs have difficulties in understanding and processing geometric figures, resulting in their performance in solving geometric problems not being better than large language models (LLMs) that only handle textual information. To solve this problem, the authors propose a new framework called "Diagram Formalization Enhanced Geometric Problem Solver" (DFE-GPS), which improves the model's understanding of geometric figures by integrating visual features, geometric formal language, and natural language representations. ### Main Issues: 1. **Insufficient understanding of geometric figures**: Most MLLMs' visual encoders are pre-trained on natural scenes, making it difficult to effectively understand geometric figures. 2. **Lack of effective geometric relationship representation methods**: Existing models lack effective methods to represent geometric relationships, leading to poor performance in solving geometric problems. ### Solutions: - **Introducing the DFE-GPS framework**: This framework includes a Diagram Formalizer module to generate formal descriptions of geometric figures, thereby enhancing the visual component and improving LLMs' recognition of geometric structures. - **Creating a large-scale geometric dataset**: The authors propose a new synthetic data generation method, creating a large-scale dataset SynthGeo228K containing 228,000 geometric figures, each annotated with formal and natural language descriptions. - **Multi-stage training process**: Through a three-stage training process, the model's performance in solving geometric problems is gradually improved. These three stages focus on training the Diagram Formalizer module, aligning visual features with the language model, and instruction fine-tuning. ### Experimental Results: - **Performance improvement**: Experimental results show that DFE-GPS performs excellently on multiple metrics, especially in terms of accuracy and process evaluation scores on multiple-choice and open-ended questions. - **Comparison with other models**: DFE-GPS significantly outperforms other existing MLLMs and LLMs, particularly in handling geometric figures. ### Conclusion: By introducing the DFE-GPS framework and a large-scale synthetic dataset, the authors successfully improved the performance of multimodal large language models in solving geometric problems, particularly addressing the shortcomings of existing models in understanding and processing geometric figures. Future work can further explore reinforcement learning and tree search strategies to further enhance the model's performance.

Diagram Formalization Enhanced Multi-Modal Geometry Problem Solver

GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training

FormalGeo: An Extensible Formalized Framework for Olympiad Geometric Problem Solving

A Symbolic Character-Aware Model for Solving Geometry Problems

A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram

FormalGeo: The First Step Toward Human-like IMO-level Geometric Automated Reasoning

Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning

G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model

EAGLE: Elevating Geometric Reasoning through LLM-empowered Visual Instruction Tuning

GOLD: Geometry Problem Solver with Natural Language Description

Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning

GeoDRL: A Self-Learning Framework for Geometry Problem Solving using Reinforcement Learning in Deductive Reasoning

Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models

FGeo-DRL: Deductive Reasoning for Geometric Problems through Deep Reinforcement Learning

MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

Fuse, Reason and Verify: Geometry Problem Solving with Parsed Clauses from Diagram

GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning

Enhancing Geometry Problem Solving with Attention Mechanism and Super-Resolution

A Deep Reinforcement Learning Agent for Geometry Online Tutoring

Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions

Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring