Abstract:Handwritten Mathematical Expression Recognition (HMER) plays a critical role in various applications, such as digitized education and scientific research. Although existing methods have achieved promising performance on publicly available datasets, they still struggle to recognize multi-line mathematical expressions (MEs), suffering from complex structures and slow inference speed. To address these issues, we propose a Line-Aware Semi-autoregressive Transformer (LAST) that treats multi-line mathematical expression sequences as two-dimensional dual-end structures. The proposed LAST utilizes a line-wise dual-end decoding strategy to decode multi-line mathematical expressions in parallel and perform dual-end decoding within each line. Specifically, we introduce a line-aware positional encoding module and a line-partitioned dual-end mask to endow LAST with line order awareness and directionality. Additionally, we adopt a shared-task optimization strategy to train LAST in both autoregressive and semi-autoregressive tasks. To evaluate the effectiveness of our approach in real-world scenarios, we have built a new Multi-line Mathematical Expression dataset (M2E), which, to the best of our knowledge, is the first of its kind and boasts with the largest character category, the largest samples of characters, and the longest average sequence length, compared to existing ME datasets. Experimental results on both the M2E dataset and publicly available datasets demonstrate the effectiveness of our proposed method. Notably, our semi-autoregressive decoding approach achieves significantly faster decoding speeds while still achieving state-of-the-art performance compared to the existing methods.

Robust Encoder-Decoder Learning Framework for Offline Handwritten Mathematical Expression Recognition Based on a Multi-Scale Deep Neural Network

Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition.

Watch, Attend and Parse: an End-to-end Neural Network Based Approach to Handwritten Mathematical Expression Recognition.

Synthetic Data Generation and Shuffled Multi-Round Training Based Offline Handwritten Mathematical Expression Recognition

Tree-based Data Augmentation and Mutual Learning for Offline Handwritten Mathematical Expression Recognition

A Deep Learning Technology based OCR Framework for Recognition Handwritten Expression and Text

Handwritten Mathematical Expression Recognition via Attention Aggregation Based Bi-directional Mutual Learning

A GRU-based Encoder-Decoder Approach with Attention for Online Handwritten Mathematical Expression Recognition

Relative Position Embedding Asymmetric Siamese Network for Offline Handwritten Mathematical Expression recognition.

Offline Handwritten Mathematical Recognition using Adversarial Learning and Transformers

Read Ten Lines at One Glance: Line-Aware Semi-Autoregressive Transformer for Multi-Line Handwritten Mathematical Expression Recognition

Deep Learning Architecture for Off-Line Recognition of Handwritten Math Symbols

ConvMath : A Convolutional Sequence Network for Mathematical Expression Recognition

DGNet: A Handwritten Mathematical Formula Recognition Network Based on Deformable Convolution and Global Context Attention

Multi-modal Attention Network for Handwritten Mathematical Expression Recognition.

Offline handwritten mathematical expression recognition with graph encoder and transformer decoder

Robust Math Formula Recognition in Degraded Chinese Document Images

Multimodal Dependence Attention and Large-Scale Data Based Offline Handwritten Formula Recognition

Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer

Offline Handwritten Mathematical Expression Evaluator Using Convolutional Neural Network

SRD: A Tree Structure Based Decoder for Online Handwritten Mathematical Expression Recognition.