Abstract:Handwritten Mathematical Expression Recognition (HMER) has wide applications in human-machine interaction scenarios, such as digitized education and automated offices. Recently, sequence-based models with encoder-decoder architectures have been commonly adopted to address this task by directly predicting LaTeX sequences of expression images. However, these methods only implicitly learn the syntax rules provided by LaTeX, which may fail to describe the position and hierarchical relationship between symbols due to complex structural relations and diverse handwriting styles. To overcome this challenge, we propose a position forest transformer (PosFormer) for HMER, which jointly optimizes two tasks: expression recognition and position recognition, to explicitly enable position-aware symbol feature representation learning. Specifically, we first design a position forest that models the mathematical expression as a forest structure and parses the relative position relationships between symbols. Without requiring extra annotations, each symbol is assigned a position identifier in the forest to denote its relative spatial position. Second, we propose an implicit attention correction module to accurately capture attention for HMER in the sequence-based decoder architecture. Extensive experiments validate the superiority of PosFormer, which consistently outperforms the state-of-the-art methods 2.03%/1.22%/2.00%, 1.83%, and 4.62% gains on the single-line CROHME 2014/2016/2019, multi-line M2E, and complex MNE datasets, respectively, with no additional latency or computational cost. Code is available at <a class="link-external link-https" href="https://github.com/SJTU-DeepVisionLab/PosFormer" rel="external noopener nofollow">this https URL</a>.

Structure-aware Mathematical Expression Recognition with Sequence-Level Modeling

Specialized Mathematical Solving by a Step-By-Step Expression Chain Generation

An Improved Approach Based on CNN-RNNs for Mathematical Expression Recognition

EDSL: An Encoder-Decoder Architecture with Symbol-Level Features for Printed Mathematical Expression Recognition

TRMER: Transformer-Based End to End Printed Mathematical Expression Recognition.

MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition

Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition

Handwritten Mathematical Expression Recognition via Attention Aggregation Based Bi-directional Mutual Learning

TAMER: Tree-Aware Transformer for Handwritten Mathematical Expression Recognition

Improving Handwritten Mathematical Expression Recognition Via Similar Symbol Distinguishing

Symbol Location-Aware Network for Improving Handwritten Mathematical Expression Recognition

ConvMath : A Convolutional Sequence Network for Mathematical Expression Recognition

A System for Recognizing Online Handwritten Mathematical Expressions and Improvement of Structure Analysis

Imbalanced Conditional Conv-Transformer for Mathematical Expression Recognition

PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer

UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition

An End-to-End Formula Recognition Method Integrated Attention Mechanism

Robust Math Formula Recognition in Degraded Chinese Document Images

Enhancing Complex Formula Recognition with Hierarchical Detail-Focused Network

A tree-BLSTM-based recognition system for online handwritten mathematical expressions

Read Ten Lines at One Glance: Line-Aware Semi-Autoregressive Transformer for Multi-Line Handwritten Mathematical Expression Recognition