Abstract:Formula recognition endeavors to automatically identify mathematical formulas from images. Currently, the Encoder-Decoder model has significantly advanced the translation from image to corresponding formula markups. Nonetheless, previous research primarily concentrated on single-line formula recognition, ignoring the recognition of multi-line formulas, which presents additional challenges such as more stringent grammatical restrictions and two- dimensional positions. In this work, we present GAP (Grammar And Position-Aware formula recognition), a comprehensive framework designed to tackle the challenges in multi-line mathematical formula recognition. First, to overcome the limitations imposed by grammar, we design a novel Grammar Aware Contrastive Learning (GACL) module, integrating complex grammar rules into the transcription model through a contrastive learning mechanism. Furthermore, primitive contrastive learning lacks clear directions for comprehending grammar rules and can lead to unstable convergence or prolonged training cycles. To enhance training efficiency, we propose Rank-Based Sampling (RBS) specialized for multi-line formulas, which guides the learning process by the importance ranking of different grammar errors. Finally, spatial location information is critical considering the two-dimensional nature of multi-line formulas. To aid the model in keeping track of that global information, we introduced a Visual Coverage (VC) mechanism that incorporates historical attention information into the image features via a parameter-free way. To validate the effectiveness of our GAP framework, we construct a new dataset Multi-Line containing 12,002 multi-line formulas and conduct extensive experiments to show the efficacy of our GAP framework in capturing grammatical rules, enhancing recognition accuracy, and enhancing training efficiency. Codes and datasets are available at https://github.com/Sinon02/GAP.

Formulaic language identification model based on GCN fusing associated information

Formula Citation Graph Based Mathematical Information Retrieval

Graph convolutional network with interactive memory fusion for aspect-based sentiment analysis

Natural Language Inference Using Lstm Model With Sentence Fusion

Use of Formulaic Language as a Predictor of L2 Oral and Written Performance

Idiomatic Expression Identification using Semantic Compatibility

The Psychological Reality of Formulaic Language

Learning to Generate CGs from Domain Specific Sentences

A Synergetic Approach to the Relationship between the Length and Frequency of Chinese Formulaic Sequences

Incorporating Deep Syntactic and Semantic Knowledge for Chinese Sequence Labeling with GCN

FSS-GCN: A graph convolutional networks with fusion of semantic and structure for emotion cause analysis

F5C-finder: An Explainable and Ensemble Biological Language Model for Predicting 5-Formylcytidine Modifications on mRNA

Modeling multiple latent information graph structures via graph convolutional network for aspect-based sentiment analysis

GAP: A Grammar and Position-Aware Framework for Efficient Recognition of Multi-Line Mathematical Formulas.

Associated words recognition in Chinese compound sentences based on deep learning

SLFNet: Generating Semantic Logic Forms from Natural Language Using Semantic Probability Graphs

GOFA: A Generative One-For-All Model for Joint Graph Language Modeling

Formula graph self-attention network for representation-domain independent materials discovery

Twain-GCN: twain-syntax graph convolutional networks for aspect-based sentiment analysis

A Method of Automatic Recognition of Attributive Clauses in Chinese Language

Generating Informative Responses with Controlled Sentence Function.