Abstract:The recent explosive interest on transformers has suggested their potential to become powerful "universal" models for computer vision tasks, such as classification, detection, and segmentation. While those attempts mainly study the discriminative models, we explore transformers on some more notoriously difficult vision tasks, e.g., generative adversarial networks (GANs). Our goal is to conduct the first pilot study in building a GAN completely free of convolutions, using only pure transformer-based architectures. Our vanilla GAN architecture, dubbed TransGAN, consists of a memory-friendly transformer-based generator that progressively increases feature resolution, and correspondingly a multi-scale discriminator to capture simultaneously semantic contexts and low-level textures. On top of them, we introduce the new module of grid self-attention for alleviating the memory bottleneck further, in order to scale up TransGAN to high-resolution generation. We also develop a unique training recipe including a series of techniques that can mitigate the training instability issues of TransGAN, such as data augmentation, modified normalization, and relative position encoding. Our best architecture achieves highly competitive performance compared to current state-of-the-art GANs using convolutional backbones. Specifically, TransGAN sets new state-of-the-art inception score of 10.43 and FID of 18.28 on STL-10, outperforming StyleGAN-V2. When it comes to higher-resolution (e.g. 256 x 256) generation tasks, such as on CelebA-HQ and LSUN-Church, TransGAN continues to produce diverse visual examples with high fidelity and impressive texture details. In addition, we dive deep into the transformer-based generation models to understand how their behaviors differ from convolutional ones, by visualizing training dynamics. The code is available at https://github.com/VITA-Group/TransGAN.

VGGAN: Visual Grounding GAN Using Panoptic Transformers

Virtual View Generation Based on 3D-Dense-attentive GAN Networks

TransVG: End-to-End Visual Grounding with Transformers

CogView: Mastering Text-to-Image Generation via Transformers

TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer

CT-GAN: A conditional Generative Adversarial Network of transformer architecture for text-to-image

SAW-GAN: Multi-granularity Text Fusion Generative Adversarial Networks for text-to-image generation

ViTGAN: Training GANs with Vision Transformers

VITAL: A Visual Interpretation on Text with Adversarial Learning for Image Labeling

The Nuts and Bolts of Adopting Transformer in GANs

Transformer-based Visual Grounding with Cross-modality Interaction

Tf-Gan: Text Feature Fusion Gan for Text-to-Image Generation

TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

Hierarchical Cross-Modal Contextual Attention Network for Visual Grounding

GLIGEN: Open-Set Grounded Text-to-Image Generation

MGF-GAN: Multi Granularity Text Feature Fusion for Text-guided-Image Synthesis

ARRPNGAN: Text-to-image GAN with attention regularization and region proposal networks

A Novel GAN Based on Progressive Growing Transformer with Capsule Embedding.

CgT-GAN: CLIP-guided Text GAN for Image Captioning

Recent Advances of Generative Adversarial Networks in Computer Vision

GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection