Abstract:The most advanced diffusion models have recently adopted increasingly deep stacked networks (e.g., U-Net or Transformer) to promote the generative emergence capabilities of vision generation models similar to large language models (LLMs). However, progressively deeper stacked networks will intuitively cause numerical propagation errors and reduce noisy prediction capabilities on generative data, which hinders massively deep scalable training of vision generation models. In this paper, we first uncover the nature that neural networks being able to effectively perform generative denoising lies in the fact that the intrinsic residual unit has consistent dynamic property with the input signal's reverse diffusion process, thus supporting excellent generative abilities. Afterwards, we stand on the shoulders of two common types of deep stacked networks to propose a unified and massively scalable Neural Residual Diffusion Models framework (Neural-RDM for short), which is a simple yet meaningful change to the common architecture of deep generative networks by introducing a series of learnable gated residual parameters that conform to the generative dynamics. Experimental results on various generative tasks show that the proposed neural residual models obtain state-of-the-art scores on image's and video's generative benchmarks. Rigorous theoretical proofs and extensive experiments also demonstrate the advantages of this simple gated residual mechanism consistent with dynamic modeling in improving the fidelity and consistency of generated content and supporting large-scale scalable training. Code is available at <a class="link-external link-https" href="https://github.com/Anonymous/Neural-RDM" rel="external noopener nofollow">this https URL</a>.

Exploring Generative Perspective of Convolutional Neural Networks by Learning Random Field Models

Learning FRAME Models Using CNN Filters

Learning FRAME Models Using CNN Filters for Knowledge Visualization.

A Theory of Generative ConvNet

Sparse and Deep Generalizations of the FRAME Model

Learning Generative ConvNets Via Multi-grid Modeling and Sampling

Generative Models for Stochastic Processes Using Convolutional Neural Networks

Learning to Generate Chairs, Tables and Cars with Convolutional Networks

Self-supervised Learning and Prediction of Microstructure Evolution with Convolutional Recurrent Neural Networks

Learning Generative Models of Scene Features

Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet.

Generative Hierarchical Learning of Sparse FRAME Models

Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patterns

Generative convolution layer for image generation

A Powerful Generative Model Using Random Weights for the Deep Image Representation

Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimization

Markov Random Field Based Convolutional Neural Networks for Image Classification

Conditional Generative ConvNets for Exemplar-Based Texture Synthesis

Neural Residual Diffusion Models for Deep Scalable Vision Generation

Generative Adversarial Nets

Generative Modeling by Inclusive Neural Random Fields with Applications in Image Generation and Anomaly Detection