Abstract:Recently, several studies proposed methods to utilize some classes of optimization problems in designing deep neural networks to encode constraints that conventional layers cannot capture. However, these methods are still in their infancy and require special treatments, such as analyzing the KKT condition, for deriving the backpropagation formula. In this paper, we propose a new layer formulation called the fixed-point iteration (FPI) layer that facilitates the use of more complicated operations in deep networks. The backward FPI layer is also proposed for backpropagation, which is motivated by the recurrent back-propagation (RBP) algorithm. But in contrast to RBP, the backward FPI layer yields the gradient by a small network module without an explicit calculation of the Jacobian. In actual applications, both the forward and backward FPI layers can be treated as nodes in the computational graphs. All components in the proposed method are implemented at a high level of abstraction, which allows efficient higher-order differentiations on the nodes. In addition, we present two practical methods of the FPI layer, FPI_NN and FPI_GD, where the update operations of FPI are a small neural network module and a single gradient descent step based on a learnable cost function, respectively. FPI\_NN is intuitive, simple, and fast to train, while FPI_GD can be used for efficient training of energy networks that have been recently studied. While RBP and its related studies have not been applied to practical examples, our experiments show the FPI layer can be successfully applied to real-world problems such as image denoising, optical flow, and multi-label classification.

A New Layer-Wise Linearized Algorithm for Feedforward Neural Network

Dynamic node creation and fast learning algorithm for a hybrid feedforward neural network

Fast Learning Algorithms for Feedforward Neural Networks

A Fast Learning Algorithm of Neural Networks by Changing Error Functions

New Dynamical Optimal Learning for Linear Multilayer FNN

A Fast Learning Algorithm of Feedforward Neural Networks by Using Novel Error Functions

Layer Collaboration in the Forward-Forward Algorithm

A Fast Hybrid Algorithm Of Global Optimization For Feedforward Neural Networks

Train Feedfoward Neural Network with Layer-wise Adaptive Rate via Approximating Back-matching Propagation

Research of New Learning Method of Feedforward Neural Network

An Improved Genetic Algorithm for Training Layered Feedforward Neural Networks

A Fast Feedforward Small-World Neural Network for Nonlinear System Modeling

Optimal filtering algorithms for fast learning in feedforward neural networks

Layer-wise Feedback Propagation

Fast Computational Approach to the Levenberg-Marquardt Algorithm for Training Feedforward Neural Networks

Analysis of Boundedness and Convergence of Online Gradient Method for Two-Layer Feedforward Neural Networks

A Layer-Wise Theoretical Framework for Deep Learning of Convolutional Neural Networks

Optimal parameters selection of back propagation algorithm in the feedforward neural network

Speeding Up the Training Process of the Mfnn by Optimizing the Hidden Layers' Outputs

Dynamical Optimal Learning For Fnn And Its Applications

Differentiable Forward and Backward Fixed-Point Iteration Layers