Abstract:Handwritten mathematical expression recognition (HMER) poses a formidable challenge due to the intricate two-dimensional structures and diverse handwriting styles. This paper introduces a novel approach to improve HMER accuracy by employing an integrated, high-capacity architecture that combines Transformer and Convolutional Neural Network (CNN) models, along with a denoising diffusion probabilistic model (DDPM)-based data augmentation technique. We explore three combination strategies for an attention-based encoder-decoder (AED) HMER model: 1) The "Tandem" strategy, which harnesses CNN features within a Transformer encoder to capture global interdependencies; 2) The "Parallel" strategy, which integrates Transformer encoder outputs with CNN outputs to achieve comprehensive feature fusion; 3) The "Mixing" strategy, which introduces multi-head self-attention (MHSA) at the final stage of the CNN. We evaluate these methods using the CROHME benchmark dataset and conduct a detailed comparative analysis. All three approaches significantly enhance model performance. Notably, the "Tandem" approach achieves expression recognition rates (ExpRate) of 54.85% and 58.56% on the CROHME 2016 and 2019 test sets, respectively, while the "Parallel" method attains 55.63% and 57.39% on the same test sets. Furthermore, we introduce an innovative data augmentation approach that utilizes DDPM to generate synthetic training samples. The DDPM, conditioned on LaTeX-rendered images, bridges the gap between printed and handwritten expressions, enabling the creation of realistic, stylistically diverse handwriting samples. This augmentation boosts the ExpRates of all strategies on both CROHME 2016 and 2019 test sets, yielding improvements of 1.6-4.6% relative to the unaugmented dataset.

An Approach Based on Transformer and Deformable Convolution for Realistic Handwriting Samples Generation

Disentangling Writer and Character Styles for Handwriting Generation

Handwriting recognition training and synthesis

Automatic Generation of Personal Chinese Handwriting by Capturing the Characteristics of Personal Handwriting.

Learning-Based Cursive Handwriting Synthesis

Handwritten Chinese Font Generation with Collaborative Stroke Refinement.

A New Approach for Synthesis and Recognition of Large Scale Handwritten Chinese Words

An Approach to Natural Stroke Extraction for Off-Line Loosely-Constrained Handwritten Chinese Characters

Deformation Transformation for Handwritten Chinese Character Shape Correction

EasyFont

Automatic Generation of Personalized Chinese Handwriting Characters

Style-preserving English Handwriting Synthesis

Transformer-based stroke relation encoding for online handwriting and sketches

One-Shot Diffusion Mimicker for Handwritten Text Generation

A Learning-based Approach to Cursive Handwriting Synthesis ?

Data Incubation -- Synthesizing Missing Data for Handwriting Recognition

Improving Handwritten Mathematical Expression Recognition via Integrating Convolutional Neural Network With Transformer and Diffusion-Based Data Augmentation

Boosting Modern and Historical Handwritten Text Recognition with Deformable Convolutions

Automatic generation of large-scale handwriting fonts via style learning.

Fully Convolutional Networks for Handwriting Recognition