Abstract:The human brain exhibits a strong ability to spontaneously associate different visual attributes of the same or similar visual scene, such as associating sketches and graffiti with real-world visual objects, usually without supervising information. In contrast, in the field of artificial intelligence, controllable generation methods like ControlNet heavily rely on annotated training datasets such as depth maps, semantic segmentation maps, and poses, which limits the method's scalability. Inspired by the neural mechanisms that may contribute to the brain's associative power, specifically the cortical modularization and hippocampal pattern completion, here we propose a self-supervised controllable generation (SCG) framework. Firstly, we introduce an equivariant constraint to promote inter-module independence and intra-module correlation in a modular autoencoder network, thereby achieving functional specialization. Subsequently, based on these specialized modules, we employ a self-supervised pattern completion approach for controllable generation training. Experimental results demonstrate that the proposed modular autoencoder effectively achieves functional specialization, including the modular processing of color, brightness, and edge detection, and exhibits brain-like features including orientation selectivity, color antagonism, and center-surround receptive fields. Through self-supervised training, associative generation capabilities spontaneously emerge in SCG, demonstrating excellent generalization ability to various tasks such as associative generation on painting, sketches, and ancient graffiti. Compared to the previous representative method ControlNet, our proposed approach not only demonstrates superior robustness in more challenging high-noise scenarios but also possesses more promising scalability potential due to its self-supervised manner.

Learning to Generate Chairs, Tables and Cars with Convolutional Networks

3D-Aware Image Synthesis Via Learning Structural and Textural Representations

Generative Creativity: Adversarial Learning For Bionic Design

Unsupervised Image-to-Image Translation with Generative Adversarial Networks.

Multi-view 3D Models from Single Images with a Convolutional Network

Exploring Generative Perspective of Convolutional Neural Networks by Learning Random Field Models

A one-to-many conditional generative adversarial network framework for multiple image-to-image translations

GEN: Generative Equivariant Networks for Diverse Image-to-Image Translation.

Learning from Pattern Completion: Self-supervised Controllable Generation

Generative convolution layer for image generation

Automatic Image Colorization with Convolutional Neural Networks and Generative Adversarial Networks

Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Learning Adversarial 3D Model Generation with 2D Image Enhancer

Texture Networks: Feed-forward Synthesis of Textures and Stylized Images

A Powerful Generative Model Using Random Weights for the Deep Image Representation

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Deep Convolutional Networks as Models of Generalization and Blending Within Visual Creativity

Learning Generative Models of Scene Features

Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data

3inGAN: Learning a 3D Generative Model from Images of a Self-similar Scene