Abstract:Image inpainting infers the missing areas of a corrupted image according to the information of the undamaged part. Many existing image inpainting methods can generate plausible inpainted results from damaged images with the fast-developed deep-learning technology. However, they still suffer from over-smoothed textures or textural distortion in the cases of complex textural details or large damaged areas. To restore textures at a fine-grained level, we propose an image inpainting method based on a hierarchical VQ-VAE with a vector credibility mechanism. It first trains the hierarchical VQ-VAE with ground truth images to update two codebooks and to obtain two corresponding vector collections containing information on ground truth images. The two vector collections are fed to a decoder to generate the corresponding high-fidelity outputs. An encoder then is trained with the corresponding damaged image. It generates vector collections approximating the ground truth by the help of the prior knowledge provided by the codebooks. After that, the two vector collections pass through the decoder from the hierarchical VQ-VAE to produce the inpainted results. In addition, we apply a vector credibility mechanism to promote vector collections from damaged images and approximate vector collections from ground truth images. To further improve the inpainting result, we apply a refinement network, which uses residual blocks with different dilation rates to acquire both global information and local textural details. Extensive experiments conducted on several datasets demonstrate that our method outperforms the state-of-the-art ones.

Reparameterizing and dynamically quantizing image features for image generation

Taming Vector-Wise Quantization for Wide-Range Image Blending with Smooth Transition

Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization

Not All Image Regions Matter: Masked Vector Quantization for Autoregressive Image Generation

Autoregressive Image Generation using Residual Quantization

Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision Transformers

Taming Scalable Visual Tokenizer for Autoregressive Image Generation

Autoregressive Image Generation without Vector Quantization

Restructuring Vector Quantization with the Rotation Trick

Vector Quantized Wasserstein Auto-Encoder

SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization

QVRF: A Quantization-error-aware Variable Rate Framework for Learned Image Compression

VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations

Hierarchical Vector-Quantized Variational Autoencoder and Vector Credibility Mechanism for High-Quality Image Inpainting

An efficient encoding algorithm for vector quantization based on subvector technique.

VQ-NeRV: A Vector Quantized Neural Representation for Videos

Finite Scalar Quantization: VQ-VAE Made Simple

VQ-DRAW: A Sequential Discrete VAE

A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation

VAE-QWGAN: Improving Quantum GANs for High Resolution Image Generation

Generating Diverse High-Fidelity Images with VQ-VAE-2