UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction

Yuan Yuan,Jingtao Ding,Jie Feng,Depeng Jin,Yong Li

DOI: https://doi.org/10.1145/3637528.3671662

2024-07-01

Abstract:Urban spatio-temporal prediction is crucial for informed decision-making, such as traffic management, resource optimization, and emergence response. Despite remarkable breakthroughs in pretrained natural language models that enable one model to handle diverse tasks, a universal solution for spatio-temporal prediction remains challenging Existing prediction approaches are typically tailored for specific spatio-temporal scenarios, requiring task-specific model designs and extensive domain-specific training data. In this study, we introduce UniST, a universal model designed for general urban spatio-temporal prediction across a wide range of scenarios. Inspired by large language models, UniST achieves success through: (i) utilizing diverse spatio-temporal data from different scenarios, (ii) effective pre-training to capture complex spatio-temporal dynamics, (iii) knowledge-guided prompts to enhance generalization capabilities. These designs together unlock the potential of building a universal model for various scenarios Extensive experiments on more than 20 spatio-temporal scenarios demonstrate UniST's efficacy in advancing state-of-the-art performance, especially in few-shot and zero-shot prediction. The datasets and code implementation are released on <a class="link-external link-https" href="https://github.com/tsinghua-fib-lab/UniST" rel="external noopener nofollow">this https URL</a>.

Machine Learning

What problem does this paper attempt to address?

The paper aims to address the challenges in the field of urban spatiotemporal prediction, particularly to create a general model that can widely adapt to various spatiotemporal scenarios. Existing spatiotemporal prediction methods are usually customized for specific scenarios, requiring a large amount of domain-specific data for training, and struggle to maintain good generalization ability in data-scarce situations. The goal of the paper is to develop a general model named UniST to overcome these limitations.

Specifically, the study addresses the following key issues:
1. **Data diversity and format inconsistency**: Spatiotemporal data from different sources have different formats, dimensions, and coverage, making standardization and unified processing difficult.
2. **Cross-scenario generalization ability**: Existing models often perform well only in one city or on specific types of data, making it difficult to transfer the learned knowledge to other cities or different types of data.
3. **Data scarcity**: The amount of available data in many cities or domains is limited, which restricts the training and generalization ability of the models.

To solve these problems, the authors propose the UniST model, which has the following features:
- **Utilizing diverse spatiotemporal data**: Learning from datasets of multiple cities and different domains to acquire rich spatiotemporal patterns.
- **Effective pre-training strategy**: Using methods such as Masked Token Modeling (MTM) for pre-training to capture complex spatiotemporal relationships.
- **Knowledge-guided prompt learning**: Designing an innovative prompt network to identify potentially shared spatiotemporal patterns to enhance the model's adaptability to new scenarios.

In summary, the main goal of the paper is to propose a new spatiotemporal prediction model, UniST, to achieve strong adaptability and generalization performance across a wide range of spatiotemporal scenarios, particularly maintaining good prediction performance even in data-scarce situations.