Abstract:In this paper, we propose a novel stroke constrained attention network (SCAN) which treats stroke as the basic unit for encoder-decoder based online handwritten mathematical expression recognition (HMER). Unlike previous methods which use trace points or image pixels as basic units, SCAN makes full use of stroke-level information for better alignment and representation. The proposed SCAN can be adopted in both single-modal (online or offline) and multi-modal HMER. For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract point-level features from input traces in online mode and employs a CNN encoder to extract pixel-level features from input images in offline mode, then use stroke constrained information to convert them into online and offline stroke-level features. Using stroke-level features can explicitly group points or pixels belonging to the same stroke, therefore reduces the difficulty of symbol segmentation and recognition via the decoder with attention mechanism. For multi-modal HMER, other than fusing multi-modal information in decoder, SCAN can also fuse multi-modal information in encoder by utilizing the stroke based alignments between online and offline modalities. The encoder fusion is a better way for combining multi-modal information as it implements the information interaction one step before the decoder fusion so that the advantages of multiple modalities can be exploited earlier and more adequately when training the encoder-decoder model. Evaluated on a benchmark published by CROHME competition, the proposed SCAN achieves the state-of-the-art performance.

HMM-based On-Line Multi-Stroke Sketch Recognition

Adaptive Online Multi-stroke Sketch Recognition Based on Hidden Markov Model

An on-line sketch recognition algorithm for composite shape

Dynamic hand gesture recognition using hidden Markov models

On-Line Sketchy Graphic Recognition Based on Adaptive HMM

Pen-based gesture design and recognition for sketch editing

An Online Multi-Stroke Sketch Recognition Method Integrated with Stroke Segmentation

Sketch-R2CNN: an Attentive Network for Vector Sketch Recognition

2D-HMM BASED CHARACTER RECOGNITION OF THE ENGINEERING DRAWING

Online composite sketchy shape recognition based on bayesian networks

Study on User Adaptation for On-Line Sketch Recognition

On-Line Sketch Recognition Using Direction Feature

<i>Sketch-R2CNN</i>: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

A HMM Based On-line Chinese Character Recognition System and Improved Training Algorithm

Freehand Sketch Recognition Based on Temporal Stroke Information

A Structural Method for Online Sketched Symbol Recognition.

User Adaptation For Online Sketchy Shape Recognition

An Efficient Primitive-Based Method to Recognize Online Sketched Symbols with Autocompletion

Stroke Constrained Attention Network for Online Handwritten Mathematical Expression Recognition

Online Sketch Recognition Using Contexts

Online Diagramming Recognition Based on Automatic Stroke Parsing and Bayesian Classifier