Abstract:Handwritten mathematical expression recognition (HMER) poses a formidable challenge due to the intricate two-dimensional structures and diverse handwriting styles. This paper introduces a novel approach to improve HMER accuracy by employing an integrated, high-capacity architecture that combines Transformer and Convolutional Neural Network (CNN) models, along with a denoising diffusion probabilistic model (DDPM)-based data augmentation technique. We explore three combination strategies for an attention-based encoder-decoder (AED) HMER model: 1) The "Tandem" strategy, which harnesses CNN features within a Transformer encoder to capture global interdependencies; 2) The "Parallel" strategy, which integrates Transformer encoder outputs with CNN outputs to achieve comprehensive feature fusion; 3) The "Mixing" strategy, which introduces multi-head self-attention (MHSA) at the final stage of the CNN. We evaluate these methods using the CROHME benchmark dataset and conduct a detailed comparative analysis. All three approaches significantly enhance model performance. Notably, the "Tandem" approach achieves expression recognition rates (ExpRate) of 54.85% and 58.56% on the CROHME 2016 and 2019 test sets, respectively, while the "Parallel" method attains 55.63% and 57.39% on the same test sets. Furthermore, we introduce an innovative data augmentation approach that utilizes DDPM to generate synthetic training samples. The DDPM, conditioned on LaTeX-rendered images, bridges the gap between printed and handwritten expressions, enabling the creation of realistic, stylistically diverse handwriting samples. This augmentation boosts the ExpRates of all strategies on both CROHME 2016 and 2019 test sets, yielding improvements of 1.6-4.6% relative to the unaugmented dataset.

An Improved Approach Based on CNN-RNNs for Mathematical Expression Recognition

TRMER: Transformer-Based End to End Printed Mathematical Expression Recognition.

Robust Math Formula Recognition in Degraded Chinese Document Images

Imbalanced Conditional Conv-Transformer for Mathematical Expression Recognition

ConvMath : A Convolutional Sequence Network for Mathematical Expression Recognition

Watch, Attend and Parse: an End-to-end Neural Network Based Approach to Handwritten Mathematical Expression Recognition.

Handwritten Mathematical Expression Recognition via Attention Aggregation Based Bi-directional Mutual Learning

Improving Handwritten Mathematical Expression Recognition via Integrating Convolutional Neural Network With Transformer and Diffusion-Based Data Augmentation

Improving Handwritten Mathematical Expression Recognition via an Attention Refinement Network.

Improving Handwritten Mathematical Expression Recognition Via Similar Symbol Distinguishing

Structure-aware Mathematical Expression Recognition with Sequence-Level Modeling

Research on Symbol Recognition for Mathematical Expressions

Offline handwritten mathematical expression recognition based on YOLOv5s

MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition

UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition

Dual Branch Network Towards Accurate Printed Mathematical Expression Recognition

A GRU-based Encoder-Decoder Approach with Attention for Online Handwritten Mathematical Expression Recognition

An End-to-End Formula Recognition Method Integrated Attention Mechanism

EDSL: An Encoder-Decoder Architecture with Symbol-Level Features for Printed Mathematical Expression Recognition

MDA-Network: Mask and Dual Attention Network for Handwritten Mathematical Expression Recognition

Relative Position Embedding Asymmetric Siamese Network for Offline Handwritten Mathematical Expression recognition.