SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field
Chong Bao,Yinda Zhang,Bangbang Yang,Tianxing Fan,Zesong Yang,Hujun Bao,Guofeng Zhang,Zhaopeng Cui
2023-03-25
Abstract:Despite the great success in 2D editing using user-friendly tools, such as Photoshop, semantic strokes, or even text prompts, similar capabilities in 3D areas are still limited, either relying on 3D modeling skills or allowing editing within only a few categories. In this paper, we present a novel semantic-driven NeRF editing approach, which enables users to edit a neural radiance field with a single image, and faithfully delivers edited novel views with high fidelity and multi-view consistency. To achieve this goal, we propose a prior-guided editing field to encode fine-grained geometric and texture editing in 3D space, and develop a series of techniques to aid the editing process, including cyclic constraints with a proxy mesh to facilitate geometric supervision, a color compositing mechanism to stabilize semantic-driven texture editing, and a feature-cluster-based regularization to preserve the irrelevant content unchanged. Extensive experiments and editing examples on both real-world and synthetic data demonstrate that our method achieves photo-realistic 3D editing using only a single edited image, pushing the bound of semantic-driven editing in 3D real-world scenes. Our project webpage: <a class="link-external link-https" href="https://zju3dv.github.io/sine/" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition