Abstract:A deep belief network (DBN) is effective to create a powerful generative model by using training data. However, it is difficult to fast determine its optimal structure given specific applications. In this paper, a growing DBN with transfer learning (TL-GDBN) is proposed to automatically decide its structure size, which can accelerate its learning process and improve model accuracy. First, a basic DBN structure with single hidden layer is initialized and then pretrained, and the learned weight parameters are frozen. Second, TL-GDBN uses TL to transfer the knowledge from the learned weight parameters to newly added neurons and hidden layers, which can achieve a growing structure until the stopping criterion for pretraining is satisfied. Third, the weight parameters derived from pretraining of TL-GDBN are further fine-tuned by using layer-by-layer partial least square regression from top to bottom, which can avoid many problems of traditional backpropagation algorithm-based fine-tuning. Moreover, the convergence analysis of the TL-GDBN is presented. Finally, TL-GDBN is tested on two benchmark data sets and a practical wastewater treatment system. The simulation results show that it has better modeling performance, faster learning speed, and more robust structure than existing models. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</italic> —Transfer learning (TL) aims to improve training effectiveness by transferring knowledge from a source domain to target domain. This paper presents a growing deep belief network (DBN) with TL to improve the training effectiveness and determine the optimal model size. Facing a complex process and real-world workflow, DBN tends to require long time for its successful training. The proposed growing DBN with TL (TL-GDBN) accelerates the learning process by instantaneously transferring the knowledge from a source domain to each new deeper or wider substructure. The experimental results show that the proposed TL-GDBN model has a great potential to deal with complex system, especially the systems with high nonlinearity. As a result, it can be readily applicable to some industrial nonlinear systems.

Gated Transfer Network for Transfer Learning

Transferring Core Knowledge via Learngenes

Unleashing the potential of GNNs via Bi-directional Knowledge Transfer

GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning

Network Grafting: Transferring Learned Features from Trained Neural Networks

Adaptive Transfer Learning on Graph Neural Networks

Fine-Tuning Graph Neural Networks Via Graph Topology Induced Optimal Transport

Gated Transformer Networks for Multivariate Time Series Classification

Knowledge Distillation with Attention for Deep Transfer Learning of Convolutional Networks

Generalized Deep Transfer Networks for Knowledge Propagation in Heterogeneous Domains.

Features are fate: a theory of transfer learning in high-dimensional regression

Improving Heterogeneous Network Knowledge Transfer Based on the Principle of Generative Adversarial

Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model

Transfer Learning with Graph Co-Regularization.

SpotTune: Transfer Learning Through Adaptive Fine-Tuning

Pay Attention to Convolution Filters: Towards Fast and Accurate Fine-Grained Transfer Learning

Graph Transformer: Learning Better Representations for Graph Neural Networks.

Towards Understanding the Transferability of Deep Representations

Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning

TL-GDBN: Growing Deep Belief Network With Transfer Learning

Enhancing Knowledge Transfer for Task Incremental Learning with Data-free Subnetwork.