Abstract:The past neural network design has largely focused on feature representation space dimension and its capacity scaling (e.g., width, depth), but overlooked the feature interaction space scaling. Recent advancements have shown shifted focus towards element-wise multiplication to facilitate higher-dimensional feature interaction space for better information transformation. Despite this progress, multiplications predominantly capture low-order interactions, thus remaining confined to a finite-dimensional interaction space. To transcend this limitation, classic kernel methods emerge as a promising solution to engage features in an infinite-dimensional space. We introduce InfiNet, a model architecture that enables feature interaction within an infinite-dimensional space created by RBF kernel. Our experiments reveal that InfiNet achieves new state-of-the-art, owing to its capability to leverage infinite-dimensional interactions, significantly enhancing model performance.

What problem does this paper attempt to address?

This paper mainly discusses the problem of dimension expansion in the feature interaction space of neural networks. Traditional neural network designs focus on dimension and capacity expansion of feature representation space (such as width and depth), but neglect the expansion of feature interaction space. Recent studies have started to focus on element-wise multiplication to promote high-dimensional feature interaction space for improving information transformation, but these methods mainly capture low-order interactions, limited to finite-dimensional interaction space. The paper proposes a new model architecture called InfiNet, which utilizes radial basis function (RBF) kernels to enable infinite-dimensional feature interaction. Through RBF kernels, the model can greatly expand the interaction space at a minimal computational cost, achieving infinite dimensions and improving model performance. InfiNet replaces the common addition or multiplication operations with RBF kernel, utilizing the theory of infinite series expansion to prove dimension approximation while maintaining low cost. The researchers observe that although existing models improve performance through multiplication operations, these explanations ignore the underlying shared design, the element-wise multiplication operation, and fail to reveal the fundamental source of improvement. Therefore, they propose a unified perspective to rethink the scale of feature interaction, that is, the dimensionality of the feature interaction space, pointing out that this is a new direction for improving neural network performance. Experiments show that the key factor to learning better representations and improving model performance is the transition from finite-dimensional feature interaction space to infinite-dimensional space. InfiNet performs better than existing parallel networks and finite-order interaction networks on large datasets such as ImageNet classification, MS COCO detection, and ADE20K segmentation, validating its effectiveness.

Infinite-Dimensional Feature Interaction

xDeepInt: a hybrid architecture for modeling the vector-wise and bit-wise feature interactions

Infinite-dimensional feature aggregation via a factorized bilinear model

InteractionNN: A Neural Network for Learning Hidden Features in Sparse Prediction.

Feature Interaction Interpretability and Beyond

BaGFN: Broad Attentive Graph Fusion Network for High-Order Feature Interactions

Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimension

Feature Interaction based Neural Network for Click-Through Rate Prediction

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think

Super Interaction Neural Network.

Progressive Feature Interaction Search for Deep Sparse Network.

Feature Learning in Infinite-Width Neural Networks

Detecting Beneficial Feature Interactions for Recommender Systems

Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks

xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems

SPNet: Utilizing Subspace Projection to Achieve Feature Interaction for Click-Through Rate

AAIN: Attentional Aggregative Interaction Network for Deep Learning Based Recommender Systems

Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion

DFINet: Dual-way Feature Interaction Network for Long-Term Series Forecasting