Abstract:Modeling many-body systems has been a long-standing challenge in science, from classical and quantum physics to computational biology. Equivariance is a critical physical symmetry for many-body dynamic systems, which enables robust and accurate prediction under arbitrary reference transformations. In light of this, great efforts have been put on encoding this symmetry into deep neural networks, which significantly boosts the prediction performance of down-streaming tasks. Some general equivariant models which are computationally efficient have been proposed, however, these models have no guarantee on the approximation power and may have information loss. In this paper, we leverage insights from the scalarization technique in differential geometry to model many-body systems by learning the gradient vector fields, which are SE(3) and permutation equivariant. Specifically, we propose the Equivariant Vector Field Network (EVFN), which is built on a novel tuple of equivariant basis and the associated scalarization and vectorization layers. Since our tuple equivariant basis forms a complete basis, learning the dynamics with our EVFN has no information loss and no tensor operations are involved before the final vectorization, which reduces the complex optimization on tensors to a minimum. We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data, as well as the equilibrium state of small molecules (molecular conformation) evolving as a statistical mechanics system. Experimental results across multiple tasks demonstrate that our model achieves best or competitive performance on baseline models in various types of datasets.

Translation Equivariant Transformer Neural Processes

Approximately Equivariant Neural Processes

Gridded Transformer Neural Processes for Large Unstructured Spatio-Temporal Data

Transformer Neural Processes -- Kernel Regression

In-Context In-Context Learning with Transformer Neural Processes

Polar Transformer Networks

Equivariant Neural Functional Networks for Transformers

Permutation Equivariance of Transformers and Its Applications

Practical Equivariances via Relational Conditional Neural Processes

NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting

Probabilistic Spatial Transformer Networks

Rényi Neural Processes

PredFormer: Transformers Are Effective Spatial-Temporal Predictive Learners

Equivariant Transformer is all you need

Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes

Neural Functional Transformers

ITTR: Unpaired Image-to-Image Translation with Transformers

Does Transformer Interpretability Transfer to RNNs?

Optimizing Non-Autoregressive Transformers with Contrastive Learning

Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision

Equivariant Vector Field Network for Many-Body System Modeling.