Abstract:Handwritten mathematical expression recognition (HMER) poses a formidable challenge due to the intricate two-dimensional structures and diverse handwriting styles. This paper introduces a novel approach to improve HMER accuracy by employing an integrated, high-capacity architecture that combines Transformer and Convolutional Neural Network (CNN) models, along with a denoising diffusion probabilistic model (DDPM)-based data augmentation technique. We explore three combination strategies for an attention-based encoder-decoder (AED) HMER model: 1) The "Tandem" strategy, which harnesses CNN features within a Transformer encoder to capture global interdependencies; 2) The "Parallel" strategy, which integrates Transformer encoder outputs with CNN outputs to achieve comprehensive feature fusion; 3) The "Mixing" strategy, which introduces multi-head self-attention (MHSA) at the final stage of the CNN. We evaluate these methods using the CROHME benchmark dataset and conduct a detailed comparative analysis. All three approaches significantly enhance model performance. Notably, the "Tandem" approach achieves expression recognition rates (ExpRate) of 54.85% and 58.56% on the CROHME 2016 and 2019 test sets, respectively, while the "Parallel" method attains 55.63% and 57.39% on the same test sets. Furthermore, we introduce an innovative data augmentation approach that utilizes DDPM to generate synthetic training samples. The DDPM, conditioned on LaTeX-rendered images, bridges the gap between printed and handwritten expressions, enabling the creation of realistic, stylistically diverse handwriting samples. This augmentation boosts the ExpRates of all strategies on both CROHME 2016 and 2019 test sets, yielding improvements of 1.6-4.6% relative to the unaugmented dataset.

MCMC Based Generative Adversarial Networks for Handwritten Numeral Augmentation

Generative Adversarial Network for Handwritten Text

A Holistic Approach to Handwritten Numeral Pair Recognition Based on Generative Models of Numeral Pairs

Learn to Augment: Joint Data Augmentation and Network Optimization for Text Recognition

Restrained Generative Adversarial Network against Overfitting in Numeric Data Augmentation

Numeric Data Augmentation Using Structural Constraint Wasserstein Generative Adversarial Networks

Advancing Multilingual Handwritten Numeral Recognition With Attention-Driven Transfer Learning

Performance Evaluation of Deep Generative Models for Generating Hand-Written Character Images

Handwritten image augmentation

Generative Adversarial Network Models for Augmenting Digit and Character Datasets Embedded in Standard Markings on Ship Bodies

The Imaginative Generative Adversarial Network: Automatic Data Augmentation for Dynamic Skeleton-Based Hand Gesture and Human Action Recognition

Generative Adversarial Network (GAN) Based Data Augmentation for Palmprint Recognition

Fonts-2-Handwriting: A Seed-Augment-Train framework for universal digit classification

Annotated Hands for Generative Models

Statistical Modeling and Learning for Recognition-Based Handwritten Numeral String Segmentation.

Handwritten text generation and strikethrough characters augmentation

Handwritten Digits Image Generation with help of Generative Adversarial Network: Machine Learning Approach

Data Augmentation Based on Generative Adversarial Network with Mixed Attention Mechanism

GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks

Improving Handwritten Mathematical Expression Recognition via Integrating Convolutional Neural Network With Transformer and Diffusion-Based Data Augmentation

An Interpretable Generative Model for Handwritten Digit Image Synthesis