Abstract:Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This paper brings the best of both worlds to pluralistic image completion: appearance prior reconstruction with transformer and texture replenishment with CNN. The former transformer recovers pluralistic coherent structures together with some coarse textures, while the latter CNN enhances the local texture details of coarse priors guided by the high-resolution masked images. To decode diversified outputs from transformers, auto-regressive sampling is the most common method, but with extremely low efficiency. We further overcome this issue by proposing a new decoding strategy, temperature annealing probabilistic sampling (TAPS), which firstly achieves more than 70× speedup of inference at most, meanwhile maintaining the high quality and diversity of the sampled global structures. Moreover, we find the full CNN architecture will lead to suboptimal solutions for guided upsampling. To render more realistic and coherent contents, we design a novel module, named texture-aware guided attention, to concurrently consider the procedures of texture copy and generation, meanwhile raising several important modifications to solve the boundary artifacts. Through dense experiments, we found the proposed method vastly outperforms state-of-the-art methods in terms of four aspects: 1) large performance boost on image fidelity even compared to deterministic completion methods; 2) better diversity and higher fidelity for pluralistic completion; 3) exceptional generalization ability on large masks and generic dataset, like ImageNet. 4) Much higher decoding efficiency over previous auto-regressive based methods.

Bridging Global Context Interactions for High-Fidelity Image Completion

Up-to-Down Network: Fusing Multi-Scale Context for 3D Semantic Scene Completion

Bridging Global Context Interactions for High-Fidelity Pluralistic Image Completion

High-Fidelity and Efficient Pluralistic Image Completion with Transformers

CTFCD: Channel Transformer Based on Full Convolutional Decoder for Single Image Deraining

Transformer-based High-Fidelity Facial Displacement Completion for Detailed 3D Face Reconstruction

CompletionFormer: Depth Completion with Convolutions and Vision Transformers

Rethinking Feature Context in Learning Image-Guided Depth Completion

A transformer-CNN parallel network for image guided depth completion

A Transformer-Based Image-Guided Depth-Completion Model with Dual-Attention Fusion Module

Bridging partial-gated convolution with transformer for smooth-variation image inpainting

Large Occluded Human Image Completion Via Image-Prior Cooperating

Deep Fusion Network for Image Completion

Distance-based Weighted Transformer Network for Image Completion

Bridging the Gap Between Indoor Depth Completion and Masked Autoencoders

LGFCTR: Local and Global Feature Convolutional Transformer for Image Matching

A Two-stage Hybrid CNN-Transformer Network for RGB Guided Indoor Depth Completion.

Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images

MAT: Mask-Aware Transformer for Large Hole Image Inpainting

IFTSDNet: An Interact-Feature Transformer Network With Spatial Detail Enhancement Module for Change Detection

Adaptive Context-Aware Multi-Modal Network for Depth Completion