Multi-Round Region-Based Optimization for Scene Sketching

Yiqi Liang,Ying Liu,Dandan Long,Ruihui Li

2024-10-05

Abstract:Scene sketching is to convert a scene into a simplified, abstract representation that captures the essential elements and composition of the original scene. It requires semantic understanding of the scene and consideration of different regions within the scene. Since scenes often contain diverse visual information across various regions, such as foreground objects, background elements, and spatial divisions, dealing with these different regions poses unique difficulties. In this paper, we define a sketch as some sets of Bezier curves. We optimize the different regions of input scene in multiple rounds. In each round of optimization, strokes sampled from the next region can seamlessly be integrated into the sketch generated in the previous round of optimization. We propose additional stroke initialization method to ensure the integrity of the scene and the convergence of optimization. A novel CLIP-Based Semantic loss and a VGG-Based Feature loss are utilized to guide our multi-round optimization. Extensive experimental results on the quality and quantity of the generated sketches confirm the effectiveness of our method.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are two key challenges in scene sketch generation:

1. **Semantic understanding of complex scenes and processing of different regions**: Scene sketch generation requires semantic understanding of the scene and taking into account different regions within the scene. Since the scene contains diverse visual information (such as foreground objects, background elements, and spatial division), processing these different regions presents unique difficulties. Specifically, foreground objects or focal points may require more detailed depiction, while background elements can be represented more loosely.

2. **Balancing the visual effect and aesthetics of the sketch**: Existing methods usually rely on explicit sketch datasets for training, and the generated sketches are often simplified and abstract expressions with a fixed style or preset. This makes it difficult to balance the visual effect of the sketch while maintaining visual attractiveness and aesthetics.

To solve these problems, the author proposes a region - based multi - round optimization method (MROSS: Multi - Round Region - Based Optimization for Scene Sketching). This method improves scene sketch generation in the following ways:

- **Region - by - region optimization**: Optimize different regions of the input scene separately to ensure that the characteristics of each region can be accurately captured.
- **Edge - guided stroke initialization**: Use the farthest point sampling (FPS) algorithm to sample stroke positions uniformly to emphasize content information.
- **Novel loss function**: Introduce CLIP - based semantic loss and VGG - based feature loss to guide the multi - round optimization process, ensuring that the generated sketches have both semantic accuracy and retain geometric details.

Through these improvements, this method can generate high - quality scene sketches at different levels of abstraction and can flexibly adjust the level of detail in different regions.