Neurosymbolic Models for Computer Graphics

Daniel Ritchie,Paul Guerrero,R. Kenny Jones,Niloy J. Mitra,Adriana Schulz,Karl D. D. Willis,Jiajun Wu
2023-04-20
Abstract:Procedural models (i.e. symbolic programs that output visual data) are a historically-popular method for representing graphics content: vegetation, buildings, textures, etc. They offer many advantages: interpretable design parameters, stochastic variations, high-quality outputs, compact representation, and more. But they also have some limitations, such as the difficulty of authoring a procedural model from scratch. More recently, AI-based methods, and especially neural networks, have become popular for creating graphic content. These techniques allow users to directly specify desired properties of the artifact they want to create (via examples, constraints, or objectives), while a search, optimization, or learning algorithm takes care of the details. However, this ease of use comes at a cost, as it's often hard to interpret or manipulate these representations. In this state-of-the-art report, we summarize research on neurosymbolic models in computer graphics: methods that combine the strengths of both AI and symbolic programs to represent, generate, and manipulate visual data. We survey recent work applying these techniques to represent 2D shapes, 3D shapes, and materials & textures. Along the way, we situate each prior work in a unified design space for neurosymbolic models, which helps reveal underexplored areas and opportunities for future research.
Graphics
What problem does this paper attempt to address?
The problem this paper attempts to address is how to combine the advantages of neural networks (machine learning) and symbolic programs (procedural modeling) to generate and manipulate visual data. Specifically: - **Combining the advantages of both models**: The paper points out that procedural models (such as symbolic programs) have the advantages of strong interpretability, high output quality, and compact representation. However, writing new procedural models requires programming and design skills, which limits their popularity. On the other hand, learning-based methods (such as neural networks) are easy to create and have good generality, but their representations are often opaque and difficult to edit. Therefore, this paper aims to explore how to combine the advantages of these two models. - **Proposing a design space for neural-symbolic models**: The authors define a design space that categorizes how neural networks and symbolic representations can be mixed to generate visual data, and based on this, they review related research work. - **Application in different computer graphics fields**: The paper reviews recent work that applies these representations to various fields of computer graphics, including 2D shapes, 3D shapes, and material and texture modeling. In summary, the core of this paper is to explore a new way of generating visual data—by combining the power of neural networks and symbolic programs, overcoming the limitations of using each individually, and thus achieving more powerful, flexible, and easily editable visual data generation techniques.