Abstract:A deep belief network (DBN) is effective to create a powerful generative model by using training data. However, it is difficult to fast determine its optimal structure given specific applications. In this paper, a growing DBN with transfer learning (TL-GDBN) is proposed to automatically decide its structure size, which can accelerate its learning process and improve model accuracy. First, a basic DBN structure with single hidden layer is initialized and then pretrained, and the learned weight parameters are frozen. Second, TL-GDBN uses TL to transfer the knowledge from the learned weight parameters to newly added neurons and hidden layers, which can achieve a growing structure until the stopping criterion for pretraining is satisfied. Third, the weight parameters derived from pretraining of TL-GDBN are further fine-tuned by using layer-by-layer partial least square regression from top to bottom, which can avoid many problems of traditional backpropagation algorithm-based fine-tuning. Moreover, the convergence analysis of the TL-GDBN is presented. Finally, TL-GDBN is tested on two benchmark data sets and a practical wastewater treatment system. The simulation results show that it has better modeling performance, faster learning speed, and more robust structure than existing models. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</italic> —Transfer learning (TL) aims to improve training effectiveness by transferring knowledge from a source domain to target domain. This paper presents a growing deep belief network (DBN) with TL to improve the training effectiveness and determine the optimal model size. Facing a complex process and real-world workflow, DBN tends to require long time for its successful training. The proposed growing DBN with TL (TL-GDBN) accelerates the learning process by instantaneously transferring the knowledge from a source domain to each new deeper or wider substructure. The experimental results show that the proposed TL-GDBN model has a great potential to deal with complex system, especially the systems with high nonlinearity. As a result, it can be readily applicable to some industrial nonlinear systems.

A Pipelined Pre-Training Algorithm For Dbns

A distributed parallel training method of deep belief networks

GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism

PipePar: A Pipelined Hybrid Parallel Approach for Accelerating Distributed DNN Training

PipeDream: Fast and Efficient Pipeline Parallel DNN Training

BaPipe: Exploration of Balanced Pipeline Parallelism for DNN Training

FreezePipe: An Efficient Dynamic Pipeline Parallel Approach Based on Freezing Mechanism for Distributed DNN Training.

Optimized Deep Belief Networks on CUDA GPUs

DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

vPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN Training

Pipelined Backpropagation at Scale: Training Large Models without Batches

CSIMD: Cross-Search Algorithm with Improved Multi-dimensional Dichotomy for Micro-Batch-Based Pipeline Parallel Training in DNN.

Sequence training of multiple deep neural networks for better performance and faster training speed

Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster

Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines

TL-GDBN: Growing Deep Belief Network With Transfer Learning

Delayed Bottlenecking: Alleviating Forgetting in Pre-trained Graph Neural Networks

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

DBS: Dynamic Batch Size For Distributed Deep Neural Network Training

A Layer-Based Sparsification Method for Distributed DNN Training.