Abstract:In recent years, convolutional neural networks have been studied in the Fourier domain for a limited environment, where competitive results can be expected for conventional image classification tasks in the spatial domain. We present a novel efficient Fourier convolutional neural network, where a new activation function is used, the additional shift Fourier transformation process is eliminated, and the number of learnable parameters is reduced. First, the Phase Rectified Linear Unit (PhaseReLU) is proposed, which is equivalent to the Rectified Linear Unit (ReLU) in the spatial domain. Second, in the proposed Fourier network, the shift Fourier transform is removed since the process is inessential for training. Lastly, we introduce two ways of reducing the number of weight parameters in the Fourier network. The basic method is to use a three-by-three sized kernel instead of five-by-five in our proposed Fourier convolutional neural network. We use the random kernel in our efficient Fourier convolutional neural network, whose standard deviation of the Gaussian distribution is used as a weight parameter. In other words, since only two scalars for each imaginary and real component per channel are required, a very small number of parameters is applied compressively. Therefore, as a result of experimenting in shallow networks, such as LeNet-3 and LeNet-5, our method achieves competitive accuracy with conventional convolutional neural networks while dramatically reducing the number of parameters. Furthermore, our proposed Fourier network, using a basic three-by-three kernel, mostly performs with higher accuracy than traditional convolutional neural networks in shallow and deep neural networks. Our experiments represent that presented kernel methods have the potential to be applied in all architecture based on convolutional neural networks.

Variable Structure and Variable Learning Rate Fourier Neural Networks Research

Simo Fourier Neural Networks Research

Deep Learning without Global Optimization by Random Fourier Neural Networks

A structure optimisation algorithm for feedforward neural network construction

A Fourier neural operator-based lightweight machine learning framework for topology optimization

Convergence Framework of Deep Learning-based Hybrid Iterative Methods and the Application to Designing a Fourier Neural Solver for Parametric PDEs

Optimization Algorithm Inspired Deep Neural Network Structure Design

Discretization Error of Fourier Neural Operators

BP Calculation of Variable Structure Neural Network Model and Its Application

Machine Learning Optimization of Compact Finite Volume Methods on Unstructured Grids

A Nonlinear Neural Network-Assisted Variable Structure Controller For Flexible Space Structure

Research on SIMO Fourier Neural Networks Based on Least Square Method

Convergence Rates of Training Deep Neural Networks Via Alternating Minimization Methods.

A Fast Learning Algorithm of Neural Networks by Changing Error Functions

Fixed-Time Neurodynamic Optimization Algorithms and Application to Circuits Design

On-line Stable Nonlinear Modelling by Structurally Adaptive Neural Nets

A Model of Chaotic Neural Network with Applications in Optimization

Evaluation for Sortie Generation Capacity of the Carrier Aircraft Based on the Variable Structure RBF Neural Network with the Fast Learning Rate

Deep Learning Based on Fourier Convolutional Neural Network Incorporating Random Kernels

Variable learning rates kernel adaptive filter with single feedback.

Understanding Training and Generalization in Deep Learning by Fourier Analysis.