Abstract:Offline hand-drawn diagram recognition is concerned with digitizing diagrams sketched on paper or whiteboard to enable further editing. Some existing models can identify the individual objects like arrows and symbols, but they become involved in the dilemma of being unable to understand a diagram’s structure. Such a shortage may be inconvenient to digitalization or reconstruction of a diagram from its hand-drawn version. Other methods can accomplish this goal, but they live on stroke temporary information and time-consuming post-processing, which somehow hinders the practicability of these methods. Recently, Convolutional Neural Networks (CNN) have been proved that they perform the state-of-the-art across many visual tasks. In this paper, we propose DrawnNet, a unified CNN-based keypoint-based detector, for recognizing individual symbols and understanding the structure of offline hand-drawn diagrams. DrawnNet is designed upon CornerNet with extensions of two novel keypoint pooling modules which serve to extract and aggregate geometric characteristics existing in polygonal contours such as rectangle, square, and diamond within hand-drawn diagrams, and an arrow orientation prediction branch which aims to predict which direction an arrow points to through predicting arrow keypoints. We conducted wide experiments on public diagram benchmarks to evaluate our proposed method. Results show that DrawnNet achieves 2.4%, 2.3%, and 1.7% recognition rate improvements compared with the state-of-the-art methods across benchmarks of FC-A, FC-B, and FA, respectively, outperforming existing diagram recognition systems on each metric. Ablation study reveals that our proposed method can effectively enable hand-drawn diagram recognition.

SpaceGTN: A Time-Agnostic Graph Transformer Network for Handwritten Diagram Recognition and Segmentation

On-line Handwritten Recognition with Optimized GDTW-SVM Algoriyhm

Instance GNN: A Learning Framework for Joint Symbol Segmentation and Recognition in Online Handwritten Diagrams

Sketchformer++: A Hierarchical Transformer Architecture for Vector Sketch Representation

DyGAT: Dynamic Stroke Classification of Online Handwritten Documents and Sketches

DrawnNet: Offline Hand-Drawn Diagram Recognition Based on Keypoint Prediction of Aggregating Geometric Characteristics

Characters as Graphs: Interpretable Handwritten Chinese Character Recognition via Pyramid Graph Transformer

Transformer-based stroke relation encoding for online handwriting and sketches

Towards Fast, Accurate and Compact Online Handwritten Chinese Text Recognition

Characters as Graphs: Recognizing Online Handwritten Chinese Characters via Spatial Graph Convolutional Network

Stroke-based semantic segmentation for scene-level free-hand sketches

Online Gesture Recognition using Transformer and Natural Language Processing

Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches

Spatial-temporal Graph Transformer Network for Skeleton-Based Temporal Action Segmentation

Online Diagramming Recognition Based on Automatic Stroke Parsing and Bayesian Classifier

Recognition of Handwritten Chinese Text by Segmentation: A Segment-annotation-free Approach

SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

Character Queries: A Transformer-based Approach to On-Line Handwritten Character Segmentation

First Experiments On A New Online Handwritten Flowchart Database

Offline handwritten mathematical expression recognition with graph encoder and transformer decoder

GPTR: Gestalt-Perception Transformer for Diagram Object Detection