Abstract:We consider the cross-modal alignment problem for free-hand sketch. Given a sequence of stroke and a rasterized image, the objective is to enhance the performance of sketch recognition through cross-modal interactions. Existing works mostly employ simple weighted adding and concatenation for late fusion, or shallow attention layers for cross-modal alignment. Due to the high heterogeneity between sketch modalities, these methods do not capture meaningful feature representations sufficiently. In this paper, we propose a sketch recognition framework CMPS for aligning Cross-Modal Pixel-and-Stroke representation, which includes novel components, namely the Semantic-Temporal Alignment Rasterization (STAR) and Pixel-Stroke Alignment (PSA) module. STAR aligns stroke with image at the semantic and temporal levels during the rasterization preprocessing phase by utilizing color variations in the RGB space for sketch. PSA, through its pre-alignment and post-alignment, learns how to align semantic connections at both pixel and stroke levels, capturing cross-modal dependencies, rather than relying on shallow matrix operations for interaction. Additionally, we introduce a concise stroke processing network called StrokeFormer. It extracts two hierarchical features, i.e., point-level and stroke-level, based on the formation mechanism of sketch. StrokeFormer outperforms most RNN-based and CNN-based models by a significant margin. Our experimental results demonstrate that proposed CMPS achieves new state-of-the-art performance on the Google QuickDraw-414 K dataset and TU-Berlin dataset. The code is available at https://github.com/WoodratTradeCo/CMPS.

2D Freehand Sketch Labeling Using CNN and CRF

Sketch-R2CNN: an Attentive Network for Vector Sketch Recognition

<i>Sketch-R2CNN</i>: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

An on-line sketch recognition algorithm for composite shape

SFSegNet: Parse Freehand Sketches using Deep Fully Convolutional Networks

Deep Neural Networks for Free-Hand Sketch Recognition.

Segmentation of Online Freehand Sketching Based on Speed Feature

Classified Segmentation of Online Complex Freehand Sketching

Segmentation of Online Sketching Using Geometric Feature

SketchGCN: Semantic Sketch Segmentation with Graph Convolutional Networks

Stroke-based semantic segmentation for scene-level free-hand sketches

CRF Learning with CNN Features for Image Segmentation

Sketch-R2CNN : An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

Cross-Modal Pixel-and-Stroke representation aligning networks for free-hand sketch recognition

On Learning Semantic Representations for Million-Scale Free-Hand Sketches

Coupling Deep Textural and Shape Features for Sketch Recognition.

Multi-column Point-CNN for Sketch Segmentation

Training Convolutional Neural Network for Sketch Recognition on Large-Scale Dataset.

Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition

SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

Deep Self-Supervised Representation Learning for Free-Hand Sketch