Abstract:In this paper, we introduce $\text{EVL}_{\text{Gen}}$, a streamlined framework designed for the pre-training of visually conditioned language generation models with high computational demands, utilizing frozen pre-trained large language models (LLMs). The conventional approach in vision-language pre-training (VLP) typically involves a two-stage optimization process: an initial resource-intensive phase dedicated to general-purpose vision-language representation learning, focused on extracting and consolidating relevant visual features. This is followed by a subsequent phase that emphasizes end-to-end alignment between visual and linguistic modalities. Our novel one-stage, single-loss framework bypasses the computationally demanding first training stage by gradually merging similar visual tokens during training, while avoiding model collapse caused by single-stage training of BLIP-2 type models. The gradual merging process effectively condenses visual information while preserving semantic richness, resulting in rapid convergence without compromising performance. Our experimental findings demonstrate that our approach accelerates the training of vision-language models by a factor of 5 without a noticeable impact on overall performance. Furthermore, we illustrate that our models significantly narrow the performance gap to current vision-language models using only 1/10 of the data. Finally, we showcase how our image-text models can seamlessly adapt to video-conditioned language generation tasks through novel soft attentive temporal token contextualizing modules. Code is available at \url{<a class="link-external link-https" href="https://github.com/yiren-jian/EVLGen" rel="external noopener nofollow">this https URL</a>}.

Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders

Expression Conditional Gan for Facial Expression-to-Expression Translation.

PCAE: A Framework of Plug-in Conditional Auto-Encoder for Controllable Text Generation

Attack Deterministic Conditional Image Generative Models for Diverse and Controllable Generation

ConditionVideo: Training-Free Condition-Guided Text-to-Video Generation

Condition-Transforming Variational Autoencoder for Generating Diverse Short Text Conversations.

An Overview on Controllable Text Generation via Variational Auto-Encoders

Improve Diverse Text Generation by Self Labeling Conditional Variational Auto Encoder

Jointly Learning Guidance Induction and Faithful Summary Generation Via Conditional Variational Autoencoders.

Plan-CVAE - A Planning-Based Conditional Variational Autoencoder for Story Generation.

Conditional Text Generation for Harmonious Human-Machine Interaction

Boosting Variational Generative Model Via Condition Enhancing and Lexical-Editing

Condition-Transforming Variational Autoencoder for Generating Diverse Short Text Conversations

Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders

Long and Diverse Text Generation with Planning-based Hierarchical Variational Model

Variational Conditional GAN for Fine-grained Controllable Image Generation

ViCo: Plug-and-play Visual Condition for Personalized Text-to-image Generation

Encoder-Agnostic Adaptation for Conditional Language Generation

Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction

Condition-transforming Variational Autoencoder for Conversation Response Generation.

Variational Autoregressive Decoder for Neural Response Generation.