Abstract:Neural functional networks (NFNs) have recently gained significant attention due to their diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representation. Previous NFN designs often depend on permutation symmetries in neural networks' weights, which traditionally arise from the unordered arrangement of neurons in hidden layers. However, these designs do not take into account the weight scaling symmetries of $\ReLU$ networks, and the weight sign flipping symmetries of $\sin$ or $\Tanh$ networks. In this paper, we extend the study of the group action on the network weights from the group of permutation matrices to the group of monomial matrices by incorporating scaling/sign-flipping symmetries. Particularly, we encode these scaling/sign-flipping symmetries by designing our corresponding equivariant and invariant layers. We name our new family of NFNs the Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFN). Because of the expansion of the symmetries, Monomial-NFN has much fewer independent trainable parameters compared to the baseline NFNs in the literature, thus enhancing the model's efficiency. Moreover, for fully connected and convolutional neural networks, we theoretically prove that all groups that leave these networks invariant while acting on their weight spaces are some subgroups of the monomial matrix group. We provide empirical evidence to demonstrate the advantages of our model over existing baselines, achieving competitive performance and efficiency.

Group Models of Artificial Polynomial and Trigonometric Higher Order Neural Networks

Models of Artificial Multi-Polynomial Higher Order Neural Networks

Financial Data Prediction by Artificial Sine and Cosine Trigonometric Higher Order Neural Networks

Data Simulations Using Cosine and Sigmoid Higher Order Neural Networks

Data Classification Using Ultra-High Frequency SINC and Trigonometric Higher Order Neural Networks

Time Series Data Analysis by Ultra-High Frequency Trigonometric Higher Order Neural Networks

Data Pattern Recognition Based on Ultra-High Frequency Sigmoid and Trigonometric Higher Order Neural Networks

Control Signal Generator Based on Ultra-High Frequency Polynomial and Trigonometric Higher Order Neural Networks

Models of Artificial Higher Order Neural Networks

SAS Nonlinear Models or Artificial Higher Order Neural Network Nonlinear Models?

Artificial Higher Order Neural Networks for Economics and Business

Monomial Matrix Group Equivariant Neural Functional Networks

Generalized Groves of Neural Additive Models: Pursuing transparent and accurate machine learning models in finance

Face Recognition Based on Higher Order Neural Network Group-Based Adaptive Tolerance Trees

Nonlinear System Modeling Using Multiple Neural Networks

Modeling Power Systems Dynamics with Symbolic Physics-Informed Neural Networks

Higher-order Neural Additive Models: An Interpretable Machine Learning Model with Feature Interactions

Physics guided neural networks for modelling of non-linear dynamics

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

Polynomial network autoregressive models with divergent orders