Abstract:Convolutional neural networks (CNNs) have become an essential tool for solving many machine vision and machine learning problems. A major element of these networks is the convolution operator which essentially computes the inner product between a weight vector and the vectorized image patches extracted by sliding a window in the image planes of the previous layer. In this paper, we propose two classes of surrogate functions for the inner product operation inherent in the convolution operator and so attain two generalizations of the convolution operator. The first one is based on the class of positive definite kernel functions where their application is justified by the kernel trick. The second one is based on the class of similarity measures defined according to some distance function. We justify this by tracing back to the basic idea behind the neocognitron which is the ancestor of CNNs. Both of these methods are then further generalized by allowing a monotonically increasing function (possibly depending on the weight vector) to be applied subsequently. Like any trainable parameter in a neural network, the template pattern and the parameters of the kernel/distance function are trained with the back-propagation algorithm. As an aside, we use the proposed framework to justify the use of sine activation function in CNNs. Additionally, we discovered a family of generalized convolution operators which is based on the convex combination of the dot-product and the negative squared Euclidean distance functions. Our experiments on the MNIST dataset show that the performance of ordinary CNNs can be achieved by generalized CNNs based on weighted L1/L2 distances, proving the applicability of the proposed generalization of the convolutional neural networks.

The Singular Values of Convolutional Layers

Singular Values for ReLU Layers

On the Learning Dynamics of Two-layer Nonlinear Convolutional Neural Networks.

On the Number of Linear Regions of Convolutional Neural Networks

A Tour of Convolutional Networks Guided by Linear Interpreters

Flattening Singular Values of Factorized Convolution for Medical Images

On the Geometry and Optimization of Polynomial Convolutional Networks

Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers

Non-linear Convolution Filters for CNN-based Learning

Generalizing the Convolution Operator in Convolutional Neural Networks

Breaking the Curse of Dimensionality with Convex Neural Networks

Convolution in Convolution for Network in Network

Is Second-order Information Helpful for Large-scale Visual Recognition?

Seeing Convolution Through the Eyes of Finite Transformation Semigroup Theory: An Abstract Algebraic Interpretation of Convolutional Neural Networks

Group Action Equivariance And Generalized Convolution In Multi-Layer Neural Networks

Spectral Norm of Convolutional Layers with Circular and Zero Paddings

Improving Efficiency in Convolutional Neural Network with Multilinear Filters

Convolutional layers are equivariant to discrete shifts but not continuous translations

Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time

SlimConv: Reducing Channel Redundancy in Convolutional Neural Networks by Features Recombining

Algebraic Complexity and Neurovariety of Linear Convolutional Networks