Abstract:Offline hand-drawn diagram recognition is concerned with digitizing diagrams sketched on paper or whiteboard to enable further editing. Some existing models can identify the individual objects like arrows and symbols, but they become involved in the dilemma of being unable to understand a diagram’s structure. Such a shortage may be inconvenient to digitalization or reconstruction of a diagram from its hand-drawn version. Other methods can accomplish this goal, but they live on stroke temporary information and time-consuming post-processing, which somehow hinders the practicability of these methods. Recently, Convolutional Neural Networks (CNN) have been proved that they perform the state-of-the-art across many visual tasks. In this paper, we propose DrawnNet, a unified CNN-based keypoint-based detector, for recognizing individual symbols and understanding the structure of offline hand-drawn diagrams. DrawnNet is designed upon CornerNet with extensions of two novel keypoint pooling modules which serve to extract and aggregate geometric characteristics existing in polygonal contours such as rectangle, square, and diamond within hand-drawn diagrams, and an arrow orientation prediction branch which aims to predict which direction an arrow points to through predicting arrow keypoints. We conducted wide experiments on public diagram benchmarks to evaluate our proposed method. Results show that DrawnNet achieves 2.4%, 2.3%, and 1.7% recognition rate improvements compared with the state-of-the-art methods across benchmarks of FC-A, FC-B, and FA, respectively, outperforming existing diagram recognition systems on each metric. Ablation study reveals that our proposed method can effectively enable hand-drawn diagram recognition.

First Experiments On A New Online Handwritten Flowchart Database

On-line Handwritten Flowchart Recognition Based on Logical Structure and Graph Grammar

Flowmind2Digital: The First Comprehensive Flowmind Recognition and Conversion Approach

Deep handwritten diagram segmentation

SpaceGTN: A Time-Agnostic Graph Transformer Network for Handwritten Diagram Recognition and Segmentation

Instance GNN: A Learning Framework for Joint Symbol Segmentation and Recognition in Online Handwritten Diagrams

Online Diagramming Recognition Based on Automatic Stroke Parsing and Bayesian Classifier

A Structural Method for Online Sketched Symbol Recognition.

A general framework for the recognition of online handwritten graphics

FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding

An Efficient Primitive-Based Method to Recognize Online Sketched Symbols with Autocompletion

First Multi-Dimensional Evaluation of Flowchart Comprehension for Multimodal Large Language Models

Symbol detection in online handwritten graphics using Faster R-CNN

An online sketching and gesture editing system for conceptual design

An Overview of Hand-Drawn Diagram Recognition Methods and Applications

AUTOMATED RECOGNITION AND TRANSLATION SYSTEM FOR PROCESS FLOW DIAGRAM BASED ON SKETCH

DrawnNet: Offline Hand-Drawn Diagram Recognition Based on Keypoint Prediction of Aggregating Geometric Characteristics

Eyes Understand the Sketch!: Gaze-Aided Stroke Grouping of Hand-Drawn Flowcharts.

Online and Offline Handwritten Chinese Character Recognition: Benchmarking on New Databases

Multimodal Dependence Attention and Large-Scale Data Based Offline Handwritten Formula Recognition

A tree-BLSTM-based recognition system for online handwritten mathematical expressions