Abstract:Many important problems require modelling large-scale spatio-temporal datasets, with one prevalent example being weather forecasting. Recently, transformer-based approaches have shown great promise in a range of weather forecasting problems. However, these have mostly focused on gridded data sources, neglecting the wealth of unstructured, off-the-grid data from observational measurements such as those at weather stations. A promising family of models suitable for such tasks are neural processes (NPs), notably the family of transformer neural processes (TNPs). Although TNPs have shown promise on small spatio-temporal datasets, they are unable to scale to the quantities of data used by state-of-the-art weather and climate models. This limitation stems from their lack of efficient attention mechanisms. We address this shortcoming through the introduction of gridded pseudo-token TNPs which employ specialised encoders and decoders to handle unstructured observations and utilise a processor containing gridded pseudo-tokens that leverage efficient attention mechanisms. Our method consistently outperforms a range of strong baselines on various synthetic and real-world regression tasks involving large-scale data, while maintaining competitive computational efficiency. The real-life experiments are performed on weather data, demonstrating the potential of our approach to bring performance and computational benefits when applied at scale in a weather modelling pipeline.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problem of large - scale spatio - temporal data modeling, especially how to effectively process and utilize unstructured, discretely - distributed spatio - temporal data. Specifically, the paper focuses on the challenges in the field of weather forecasting, that is, how to combine unstructured data from sources such as observation stations with existing gridded data (such as meteorological model outputs) to improve prediction accuracy and reduce computational costs. #### Main problems: 1. **Limitations of existing methods**: - Existing Transformer - based methods mainly focus on processing structured grid data, ignoring a large amount of valuable unstructured observation data. - These methods face problems of high computational complexity and difficulty in scaling when dealing with large - scale spatio - temporal data. 2. **Objectives**: - Develop a model that can efficiently process large - scale unstructured spatio - temporal data. - Propose a new framework that can handle unstructured data from multiple sources while maintaining low computational complexity and perform well in practical applications. #### Solutions: The authors introduce **Gridded Transformer Neural Processes (Gridded TNPs)** and solve the problems through the following innovations: 1. **Pseudo - token Grid Encoder**: - Use "pseudo - tokens" to map unstructured data onto the grid, so that it can be processed using an efficient attention mechanism. - This method is more effective than traditional kernel interpolation methods and can better capture the spatial structure of the data. 2. **Efficient attention mechanism**: - Introduce efficient attention mechanisms such as ViT (Vision Transformer) and Swin Transformer to reduce computational complexity and enable the model to handle larger - scale datasets. 3. **Nearest - neighbour Cross - attention Decoder**: - By only allowing the nearest - neighbour pseudo - tokens to participate in cross - attention calculations, further reduce the computational complexity while improving the performance of the model. 4. **Multimodal data processing**: - Propose two methods for processing multi - source data: single - pseudo - token grid encoder and multi - pseudo - token grid encoder to adapt to different types of input data. Through these innovations, the paper demonstrates the superior performance of Gridded TNPs on synthetic data and real - world meteorological data, especially when dealing with large - scale and complex spatio - temporal data.

Gridded Transformer Neural Processes for Large Unstructured Spatio-Temporal Data

Transformer Neural Processes -- Kernel Regression

Spatio-Temporal Transformer Network for Weather Forecasting

Translation Equivariant Transformer Neural Processes

NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting

A Hierarchical Spatial Transformer for Massive Point Samples in Continuous Space

NTDformer: A Multi-Scale Forecasting Model for Non-Stationary Multilevel Time Series

Transformer-enhanced spatiotemporal neural network for post-processing of precipitation forecasts

WeatherFormer: Empowering Global Numerical Weather Forecasting with Space-Time Transformer

A Novel Multiscale Transformer Network Framework for Natural Gas Consumption Forecasting

Leveraging 2D Information for Long-term Time Series Forecasting with Vanilla Transformers

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

STC-ViT: Spatio Temporal Continuous Vision Transformer for Weather Forecasting

Interpretable Transformer Neural Network Prediction of Diverse Environmental Time Series Using Weather Forecasts

TENT: Tensorized Encoder Transformer for Temperature Forecasting

In-Context In-Context Learning with Transformer Neural Processes

GTHP: a novel graph transformer Hawkes process for spatiotemporal event prediction

Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction

Multi-scale Transformer Pyramid Networks for Multivariate Time Series Forecasting

Scaling transformer neural networks for skillful and reliable medium-range weather forecasting

PredFormer: Transformers Are Effective Spatial-Temporal Predictive Learners