Abstract:A deep belief network (DBN) is effective to create a powerful generative model by using training data. However, it is difficult to fast determine its optimal structure given specific applications. In this paper, a growing DBN with transfer learning (TL-GDBN) is proposed to automatically decide its structure size, which can accelerate its learning process and improve model accuracy. First, a basic DBN structure with single hidden layer is initialized and then pretrained, and the learned weight parameters are frozen. Second, TL-GDBN uses TL to transfer the knowledge from the learned weight parameters to newly added neurons and hidden layers, which can achieve a growing structure until the stopping criterion for pretraining is satisfied. Third, the weight parameters derived from pretraining of TL-GDBN are further fine-tuned by using layer-by-layer partial least square regression from top to bottom, which can avoid many problems of traditional backpropagation algorithm-based fine-tuning. Moreover, the convergence analysis of the TL-GDBN is presented. Finally, TL-GDBN is tested on two benchmark data sets and a practical wastewater treatment system. The simulation results show that it has better modeling performance, faster learning speed, and more robust structure than existing models. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</italic> —Transfer learning (TL) aims to improve training effectiveness by transferring knowledge from a source domain to target domain. This paper presents a growing deep belief network (DBN) with TL to improve the training effectiveness and determine the optimal model size. Facing a complex process and real-world workflow, DBN tends to require long time for its successful training. The proposed growing DBN with TL (TL-GDBN) accelerates the learning process by instantaneously transferring the knowledge from a source domain to each new deeper or wider substructure. The experimental results show that the proposed TL-GDBN model has a great potential to deal with complex system, especially the systems with high nonlinearity. As a result, it can be readily applicable to some industrial nonlinear systems.

An Efficient Deep Belief Network with Fuzzy Learning for Nonlinear System Modeling.

A Sparse Deep Belief Network with Efficient Fuzzy Learning Framework

A neurofuzzy network structure for nonlinear system modeling

An Efficient Self-Organizing Deep Fuzzy Neural Network for Nonlinear System Modeling

A Novel Nonlinear Dynamic System Modeling Approach Using Radial Basis Function Neural Networks

A Fuzzy Deep Model Based on Fuzzy Restricted Boltzmann Machines for High-dimensional Data Classification

A Self-Organizing Deep Belief Network for Nonlinear System Modeling

Sparse Bayesian Approach to Fast Learning Network for Multiclassification.

Additive Dynamic Bayesian Networks for Enhanced Feature Learning in Soft Sensor Modeling

A Fast Feedforward Small-World Neural Network for Nonlinear System Modeling

Unsupervised Feature Learning Using Markov Deep Belief Network.

An Adaptive Deep Belief Network With Sparse Restricted Boltzmann Machines

Nonlinear System Modeling by Competitive Learning and Adaptive Fuzzy Inference System.

Modeling of Nonlinear System Based on Deep Learning Framework

Data-Knowledge-Based Fuzzy Neural Network for Nonlinear System Identification

TL-GDBN: Growing Deep Belief Network With Transfer Learning

Research on the Method of Neural Network Modeling Based on FCM Algorithm and Its Application on Vision-based Sensors

Modeling and Decision-making of Financial Time Series Data with DBN

Feature Back-Tracking with Sparse Deep Belief Networks.

A Novel Neural Network Training Framework with Data Assimilation

Complex Systems Modeling Using Dynamic Fuzzy Neural Networks