Abstract:Recent work in neural networks for image classification has seen a strong tendency towards increasing the spatial context. Whether achieved through large convolution kernels or self-attention, models scale poorly with the increased spatial context, such that the improved model accuracy often comes at significant costs. In this paper, we propose a module for studying the effective filter size of convolutional neural networks. To facilitate such a study, several challenges need to be addressed: 1) we need an effective means to train models with large filters (potentially as large as the input data) without increasing the number of learnable parameters 2) the employed convolution operation should be a plug-and-play module that can replace conventional convolutions in a CNN and allow for an efficient implementation in current frameworks 3) the study of filter sizes has to be decoupled from other aspects such as the network width or the number of learnable parameters 4) the cost of the convolution operation itself has to remain manageable i.e. we cannot naively increase the size of the convolution kernel. To address these challenges, we propose to learn the frequency representations of filter weights as neural implicit functions, such that the better scalability of the convolution in the frequency domain can be leveraged. Additionally, due to the implementation of the proposed neural implicit function, even large and expressive spatial filters can be parameterized by only a few learnable weights. Our analysis shows that, although the proposed networks could learn very large convolution kernels, the learned filters are well localized and relatively small in practice when transformed from the frequency to the spatial domain. We anticipate that our analysis of individually optimized filter sizes will allow for more efficient, yet effective, models in the future.

Scaling Continuous Kernels with Sparse Fourier Domain Learning

Self-representative kernel concept factorization

How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets

Sparse multiresolution representations with adaptive kernels

A Temporal Kernel Approach for Deep Learning with Continuous-time Information

How Feature Learning Can Improve Neural Scaling Laws

Super-resolution and Robust Sparse Continuous Fourier Transform in Any Constant Dimension: Nearly Linear Time and Sample Complexity

Learning Random Fourier Features by Hybrid Constrained Optimization

A Dual-space Multilevel Kernel-splitting Framework for Discrete and Continuous Convolution

Scaling Up 3D Kernels with Bayesian Frequency Re-parameterization for Medical Image Segmentation

Deep Learning Based on Fourier Convolutional Neural Network Incorporating Random Kernels

Integrated Variational Fourier Features for Fast Spatial Modelling with Gaussian Processes

End-to-end Kernel Learning via Generative Random Fourier Features

Fast Evaluation of Additive Kernels: Feature Arrangement, Fourier Methods, and Kernel Derivatives

Isolation Kernel: The X Factor in Efficient and Effective Large Scale Online Kernel Learning

Sparsity-Aware Distributed Learning for Gaussian Processes with Linear Multiple Kernel

Sparse Gaussian Processes with Spherical Harmonic Features Revisited

Sparse Indefinite Kernel Learning on Sphere.

Plastic Learning with Deep Fourier Features

A Fourier-based Framework for Domain Generalization

As large as it gets: Learning infinitely large Filters via Neural Implicit Functions in the Fourier Domain