EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Models

Eungbean Lee,Somi Jeong,Kwanghoon Sohn

DOI: https://doi.org/10.1007/978-3-031-72624-8_18

2024-10-13

Abstract:Exemplar-guided image translation, synthesizing photo-realistic images that conform to both structural control and style exemplars, is attracting attention due to its ability to enhance user control over style manipulation. Previous methodologies have predominantly depended on establishing dense correspondences across cross-domain inputs. Despite these efforts, they incur quadratic memory and computational costs for establishing dense correspondence, resulting in limited versatility and performance degradation. In this paper, we propose a novel approach termed Exemplar-guided Image Translation with Brownian-Bridge Diffusion Models (EBDM). Our method formulates the task as a stochastic Brownian bridge process, a diffusion process with a fixed initial point as structure control and translates into the corresponding photo-realistic image while being conditioned solely on the given exemplar image. To efficiently guide the diffusion process toward the style of exemplar, we delineate three pivotal components: the Global Encoder, the Exemplar Network, and the Exemplar Attention Module to incorporate global and detailed texture information from exemplar images. Leveraging Bridge diffusion, the network can translate images from structure control while exclusively conditioned on the exemplar style, leading to more robust training and inference processes. We illustrate the superiority of our method over competing approaches through comprehensive benchmark evaluations and visual results.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively combine structural control and style examples to generate high-quality, realistic images in image translation tasks. Specifically, existing methods typically rely on establishing dense correspondences between cross-domain inputs, which leads to increased memory and computational costs, limiting the flexibility and performance of the methods. To address these issues, the authors propose a novel method—Example-Based Diffusion Model (EBDM) based on the Brownian Bridge diffusion process, which generates images directly from structural control and translates them only under the given style example conditions. This method not only improves the robustness of training and inference but also effectively captures the global and detailed texture information in style examples through components such as the global encoder, example network, and example attention module.