LFS-GAN: Lifelong Few-Shot Image Generation

Juwon Seo,Ji-Su Kang,Gyeong-Moon Park
2023-09-05
Abstract:We address a challenging lifelong few-shot image generation task for the first time. In this situation, a generative model learns a sequence of tasks using only a few samples per task. Consequently, the learned model encounters both catastrophic forgetting and overfitting problems at a time. Existing studies on lifelong GANs have proposed modulation-based methods to prevent catastrophic forgetting. However, they require considerable additional parameters and cannot generate high-fidelity and diverse images from limited data. On the other hand, the existing few-shot GANs suffer from severe catastrophic forgetting when learning multiple tasks. To alleviate these issues, we propose a framework called Lifelong Few-Shot GAN (LFS-GAN) that can generate high-quality and diverse images in lifelong few-shot image generation task. Our proposed framework learns each task using an efficient task-specific modulator - Learnable Factorized Tensor (LeFT). LeFT is rank-constrained and has a rich representation ability due to its unique reconstruction technique. Furthermore, we propose a novel mode seeking loss to improve the diversity of our model in low-data circumstances. Extensive experiments demonstrate that the proposed LFS-GAN can generate high-fidelity and diverse images without any forgetting and mode collapse in various domains, achieving state-of-the-art in lifelong few-shot image generation task. Surprisingly, we find that our LFS-GAN even outperforms the existing few-shot GANs in the few-shot image generation task. The code is available at Github.
Computer Vision and Pattern Recognition,Artificial Intelligence
What problem does this paper attempt to address?
### Problems Addressed by the Paper The paper primarily addresses the challenging task of **Lifelong Few-Shot Image Generation**. Specifically: 1. **Combining Lifelong Learning and Few-Shot Learning**: - The paper proposes a novel framework, namely **Lifelong Few-Shot Generative Adversarial Network (LFS-GAN)**, designed to continuously learn multiple tasks under limited data conditions and generate high-quality and diverse images. - In this task, the generative model needs to learn from a small number of samples in each task and retain the ability to generate images from previous tasks while continuously learning new ones. 2. **Avoiding Catastrophic Forgetting**: - Traditional methods tend to forget previously learned knowledge when learning multiple tasks sequentially. LFS-GAN effectively addresses this issue by introducing a new weight modulation technique—Learnable Factorized Tensor (LeFT). - LeFT captures task-specific knowledge at a low memory cost while freezing the learned weights, thus avoiding catastrophic forgetting. 3. **Enhancing Generative Diversity**: - To further improve the diversity of generated images, the paper proposes a new mode-seeking loss (Cluster-Wise Mode Seeking Loss), which enhances diversity by maximizing the relative distances between intermediate latent codes, feature maps, and images. - Additionally, the paper introduces a new diversity metric—Balanced Inter- and Intra-cluster LPIPS (B-LPIPS), to accurately evaluate the diversity of generated images. In summary, the paper aims to address two key challenges in the task of lifelong few-shot image generation: **catastrophic forgetting and mode collapse**, and achieves high-quality and diverse image generation through a series of innovative methods.