SketchGPT: Autoregressive Modeling for Sketch Generation and Recognition

Adarsh Tiwari,Sanket Biswas,Josep Lladós

2024-05-06

Abstract:We present SketchGPT, a flexible framework that employs a sequence-to-sequence autoregressive model for sketch generation, and completion, and an interpretation case study for sketch recognition. By mapping complex sketches into simplified sequences of abstract primitives, our approach significantly streamlines the input for autoregressive modeling. SketchGPT leverages the next token prediction objective strategy to understand sketch patterns, facilitating the creation and completion of drawings and also categorizing them accurately. This proposed sketch representation strategy aids in overcoming existing challenges of autoregressive modeling for continuous stroke data, enabling smoother model training and competitive performance. Our findings exhibit SketchGPT's capability to generate a diverse variety of drawings by adding both qualitative and quantitative comparisons with existing state-of-the-art, along with a comprehensive human evaluation study. The code and pretrained models will be released on our official GitHub.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem this paper attempts to address is the challenges in tasks such as sketch generation, completion, and recognition by introducing an autoregressive generative model named SketchGPT. Specifically, the paper aims to:

1. **Propose a GPT-based autoregressive model**: This model can learn the neural representation of sketches, capturing the sequential dependencies in their dynamic drawing process.
2. **Simplify input data**: By mapping complex sketches into simplified sequences of abstract primitives, the model's training complexity is reduced, and its generalization ability is improved.
3. **Achieve multi-task processing**: Construct a multi-task model capable of predicting the next stroke, generating, completing, and recognizing sketches, demonstrating its versatility in sketch-related tasks.
4. **Conduct quantitative and qualitative evaluations**: Validate the quality of the model-generated sketches through comparisons with existing state-of-the-art models and comprehensive human evaluation studies.

Through these methods and experiments, the paper aims to overcome the challenges faced by existing autoregressive models on continuous stroke data and provide a more flexible and efficient solution.