Abstract:Recent work in neural networks for image classification has seen a strong tendency towards increasing the spatial context. Whether achieved through large convolution kernels or self-attention, models scale poorly with the increased spatial context, such that the improved model accuracy often comes at significant costs. In this paper, we propose a module for studying the effective filter size of convolutional neural networks. To facilitate such a study, several challenges need to be addressed: 1) we need an effective means to train models with large filters (potentially as large as the input data) without increasing the number of learnable parameters 2) the employed convolution operation should be a plug-and-play module that can replace conventional convolutions in a CNN and allow for an efficient implementation in current frameworks 3) the study of filter sizes has to be decoupled from other aspects such as the network width or the number of learnable parameters 4) the cost of the convolution operation itself has to remain manageable i.e. we cannot naively increase the size of the convolution kernel. To address these challenges, we propose to learn the frequency representations of filter weights as neural implicit functions, such that the better scalability of the convolution in the frequency domain can be leveraged. Additionally, due to the implementation of the proposed neural implicit function, even large and expressive spatial filters can be parameterized by only a few learnable weights. Our analysis shows that, although the proposed networks could learn very large convolution kernels, the learned filters are well localized and relatively small in practice when transformed from the frequency to the spatial domain. We anticipate that our analysis of individually optimized filter sizes will allow for more efficient, yet effective, models in the future.

Les Houches Lectures on Deep Learning at Large & Infinite Width

Les Houches lectures on deep learning at large and infinite width *

Stochastic Neural Networks with Infinite Width are Deterministic

Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functions

Feature Learning in Infinite-Width Neural Networks

Proportional infinite-width infinite-depth limit for deep linear neural networks

Wide stable neural networks: Sample regularity, functional convergence and Bayesian inverse problems

Sparse representations, inference and learning

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training

Dynamically Stable Infinite-Width Limits of Neural Classifiers

Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers

High-dimensional learning of narrow neural networks

A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit

Infinite‐width limit of deep linear neural networks

TASI Lectures on Physics for Machine Learning

Infinite Width Graph Neural Networks for Node Regression/ Classification

Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility

Posterior Inference on Shallow Infinitely Wide Bayesian Neural Networks under Weights with Unbounded Variance

Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks

As large as it gets: Learning infinitely large Filters via Neural Implicit Functions in the Fourier Domain