Abstract:Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This paper brings the best of both worlds to pluralistic image completion: appearance prior reconstruction with transformer and texture replenishment with CNN. The former transformer recovers pluralistic coherent structures together with some coarse textures, while the latter CNN enhances the local texture details of coarse priors guided by the high-resolution masked images. To decode diversified outputs from transformers, auto-regressive sampling is the most common method, but with extremely low efficiency. We further overcome this issue by proposing a new decoding strategy, temperature annealing probabilistic sampling (TAPS), which firstly achieves more than 70× speedup of inference at most, meanwhile maintaining the high quality and diversity of the sampled global structures. Moreover, we find the full CNN architecture will lead to suboptimal solutions for guided upsampling. To render more realistic and coherent contents, we design a novel module, named texture-aware guided attention, to concurrently consider the procedures of texture copy and generation, meanwhile raising several important modifications to solve the boundary artifacts. Through dense experiments, we found the proposed method vastly outperforms state-of-the-art methods in terms of four aspects: 1) large performance boost on image fidelity even compared to deterministic completion methods; 2) better diversity and higher fidelity for pluralistic completion; 3) exceptional generalization ability on large masks and generic dataset, like ImageNet. 4) Much higher decoding efficiency over previous auto-regressive based methods.

A transformer-CNN parallel network for image guided depth completion

CompletionFormer: Depth Completion with Convolutions and Vision Transformers

A Transformer-Based Image-Guided Depth-Completion Model with Dual-Attention Fusion Module

TDCNet: Transparent Objects Depth Completion with CNN-Transformer Dual-Branch Parallel Network

SDformer: Efficient End-to-End Transformer for Depth Completion

A Two-stage Hybrid CNN-Transformer Network for RGB Guided Indoor Depth Completion.

Monocular Depth Estimation Algorithm Integrating Parallel Transformer and Multi-Scale Features

Gated Recurrent Fusion UNet for Depth Completion

Improving Depth Gradient Continuity in Transformers: A Comparative Study on Monocular Depth Estimation with CNN

RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion

Combing Transformer and CNN for Monocular Depth Estimation

CTFCD: Channel Transformer Based on Full Convolutional Decoder for Single Image Deraining

Gated Cross-Attention Network for Depth Completion

From Patch to Pixel: A Transformer-Based Hierarchical Framework for Compressive Image Sensing.

PCTNet: Depth Estimation from Single Structured Light Image with a Parallel CNN-transformer Network

PCTDepth: Exploiting Parallel CNNs and Transformer via Dual Attention for Monocular Depth Estimation

High-Fidelity and Efficient Pluralistic Image Completion with Transformers

CTA-Net: A CNN-Transformer Aggregation Network for Improving Multi-Scale Feature Extraction

Efficient Depth Completion Network Based on Dynamic Gated Fusion

PCformer: A Parallel Convolutional Transformer Network for 360° Depth Estimation

Lightweight Self-Supervised Monocular Depth Estimation Through CNN and Transformer Integration