Abstract:Our animation studio has developed a practical style transfer pipeline for creating stylized 3D animation, which is suitable for complex real-world production. This paper presents the insights from our development process, where we explored various options to balance quality, artist control, and workload, leading to several key decisions. For example, we chose patch-based texture synthesis over machine learning for better control and to avoid training data issues. We also addressed specifying style exemplars, managing multiple colors within a scene, controlling outlines and shadows, and reducing temporal noise. These insights were used to further refine our pipeline, ultimately enabling us to produce an experimental short film showcasing various styles.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to develop a practical style transfer pipeline to create 3D animations with artistic styles. Specifically, the authors aim to balance quality, artist control and workload, so that this technology can be applied in complex real - production environments. The following are the key issues mentioned in the paper:
1. **Enhancing Artistic Expression**:
- Enhance the artistic expressiveness of 3D animations through style transfer techniques, especially to simulate the unique and diverse styles of 2D hand - drawn art (such as watercolor paintings and oil paintings).
2. **Handling Complex Scenes**:
- Handle complex production scenes containing many objects and materials, bridging the gap between academic research and actual production.
3. **Artist Control**:
- Provide a high degree of artist control, for example, generate intermediate outputs that can be used in the compositing stage, rather than a fully automated black - box process.
4. **Practical Workload Management**:
- Balance quality and effort, solve the complexity in real - world production, and ensure the practical feasibility of the workflow.
To solve these problems, the authors explored multiple options and made several key decisions, such as:
- Choosing a patch - based texture synthesis method instead of machine learning to gain better control and avoid training data issues.
- Solving problems such as the specification of style examples, managing multiple colors in the scene, controlling contours and shadows, and reducing temporal noise.
These decisions and solutions ultimately enabled them to produce an experimental short film showing various styles, proving the feasibility and effectiveness of their style transfer pipeline in actual production.
### Formula Summary
The formulas involved in the paper are mainly used to describe the style transfer process. For example, the Image Analogies formula is as follows:
Given a style example \( A' \) and a guide image \( A=\{A_1,\dots,A_n\} \) (for the style example) and \( B = \{B_1,\dots,B_n\} \) (for the target scene), where \( n \) is the number of guide images, generate a stylized target image \( B' \) such that the relationship
\[ A:A'::B:B' \]
holds.
In addition, in the part of temporal noise reduction, the paper describes an additional style transfer step for advection, whose formula is defined as:
\[ \{A_1,\dots,A_n,A'\}:A'::\{B_{t,1},\dots,B_{t,n},B'_{adv}\}:B'_t \]
Here \( A \) and \( A' \) remain unchanged in all frames, while \( B_t \) is updated frame - by - frame.