Abstract:Handwritten mathematical expression recognition (HMER) poses a formidable challenge due to the intricate two-dimensional structures and diverse handwriting styles. This paper introduces a novel approach to improve HMER accuracy by employing an integrated, high-capacity architecture that combines Transformer and Convolutional Neural Network (CNN) models, along with a denoising diffusion probabilistic model (DDPM)-based data augmentation technique. We explore three combination strategies for an attention-based encoder-decoder (AED) HMER model: 1) The "Tandem" strategy, which harnesses CNN features within a Transformer encoder to capture global interdependencies; 2) The "Parallel" strategy, which integrates Transformer encoder outputs with CNN outputs to achieve comprehensive feature fusion; 3) The "Mixing" strategy, which introduces multi-head self-attention (MHSA) at the final stage of the CNN. We evaluate these methods using the CROHME benchmark dataset and conduct a detailed comparative analysis. All three approaches significantly enhance model performance. Notably, the "Tandem" approach achieves expression recognition rates (ExpRate) of 54.85% and 58.56% on the CROHME 2016 and 2019 test sets, respectively, while the "Parallel" method attains 55.63% and 57.39% on the same test sets. Furthermore, we introduce an innovative data augmentation approach that utilizes DDPM to generate synthetic training samples. The DDPM, conditioned on LaTeX-rendered images, bridges the gap between printed and handwritten expressions, enabling the creation of realistic, stylistically diverse handwriting samples. This augmentation boosts the ExpRates of all strategies on both CROHME 2016 and 2019 test sets, yielding improvements of 1.6-4.6% relative to the unaugmented dataset.

Handwritten image augmentation

Handwritten text generation and strikethrough characters augmentation

Learn to Augment: Joint Data Augmentation and Network Optimization for Text Recognition

Complex Wavelet SSIM based Image Data Augmentation

Enhancing Performance of Deep Learning Models with a Novel Data Augmentation Approach

Data augmentation techniques based on deep learning for Chinese paintings

PairAug: What Can Augmented Image-Text Pairs Do for Radiology?

Image Data Augmentation for Deep Learning: A Survey

A survey on Image Data Augmentation for Deep Learning

A Study of Augmentation Methods for Handwritten Stenography Recognition

Soft Augmentation for Image Classification

Cap2Aug: Caption guided Image to Image data Augmentation

Improving Handwritten Mathematical Expression Recognition via Integrating Convolutional Neural Network With Transformer and Diffusion-Based Data Augmentation

Text Data Augmentation for Deep Learning

Augment the Pairs: Semantics-Preserving Image-Caption Pair Augmentation for Grounding-Based Vision and Language Models

Augmentation Inside the Network

KeepAugment: A Simple Information-Preserving Data Augmentation Approach

Style Augmentation: Data Augmentation via Style Randomization

Data-Efficient Augmentation for Training Neural Networks

Enhancing skin lesion diagnosis with data augmentation techniques: a review of the state-of-the-art

Handwritten Mathematical Expression Recognition: An approach on data augmentation