Abstract:Consider learning a generative model for time-series data. The sequential
setting poses a unique challenge: Not only should the generator capture the
conditional dynamics of (stepwise) transitions, but its open-loop rollouts
should also preserve the joint distribution of (multi-step) trajectories. On
one hand, autoregressive models trained by MLE allow learning and computing
explicit transition distributions, but suffer from compounding error during
rollouts. On the other hand, adversarial models based on GAN training alleviate
such exposure bias, but transitions are implicit and hard to assess. In this
work, we study a generative framework that seeks to combine the strengths of
both: Motivated by a moment-matching objective to mitigate compounding error,
we optimize a local (but forward-looking) transition policy, where the
reinforcement signal is provided by a global (but stepwise-decomposable) energy
model trained by contrastive estimation. At training, the two components are
learned cooperatively, avoiding the instabilities typical of adversarial
objectives. At inference, the learned policy serves as the generator for
iterative sampling, and the learned energy serves as a trajectory-level measure
for evaluating sample quality. By expressly training a policy to imitate
sequential behavior of time-series features in a dataset, this approach
embodies "generation by imitation". Theoretically, we illustrate the
correctness of this formulation and the consistency of the algorithm.
Empirically, we evaluate its ability to generate predictively useful samples
from real-world datasets, verifying that it performs at the standard of
existing benchmarks.
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve
This paper aims to address key challenges in time series data generation. Specifically, the authors focus on how to design a generative model that can capture both the conditional dynamics at each step of the time series (i.e., stepwise transitions) and maintain the joint distribution of multi-step trajectories. Both aspects are crucial in time series generation:
1. **Accuracy of Conditional Dynamics**: A good generator should accurately capture the conditional dynamics at each step of the time series \( p(x_t | x_1, \ldots, x_{t-1}) \). This is important because the effectiveness of any downstream time series analysis depends on these correlations between time dimensions and feature dimensions.
2. **Joint Distribution of Multi-step Trajectories**: When the generator unfolds recursively, it should also respect the joint distribution of multi-step trajectories \( p(x_1, \ldots, x_T) \). This is equally important because if the synthetic trajectories inadvertently fall outside the support of the original data, they will be useless.
### Limitations of Existing Methods
Existing time series generation methods mainly fall into two categories, each with its own limitations:
1. **Autoregressive Models**: Autoregressive models trained via Maximum Likelihood Estimation (MLE) can directly learn and compute explicit transition distributions, but they are prone to error accumulation during multi-step generation. This is because the model is trained in a closed-loop manner based on real inputs, while inference is done in an open-loop manner based on its previous outputs.
2. **Generative Adversarial Networks (GANs)**: GAN-based adversarial models can alleviate exposure bias issues, but the transitions are implicit and hard to evaluate. Moreover, the adversarial objective function leads to an unstable optimization process, especially on time series data.
### Proposed Solution in the Paper
To combine the strengths of the above two methods, the authors propose a new generative framework called "Time-series Generation by Contrastive Imitation" (TimeGCI). The main features of this framework include:
1. **Local (Forward) Transition Policy**: A global energy model trained via contrastive estimation provides reinforcement signals to optimize a local (but forward) transition policy. This ensures that the generator can capture the conditional dynamics of the time series while avoiding the instability brought by adversarial objective functions.
2. **Global (Stepwise Decomposable) Energy Model**: The energy model is used to evaluate sample quality, ensuring that the generated trajectories conform to the joint distribution of the original data.
3. **Non-adversarial Training**: The generator and evaluator learn collaboratively during the training process, avoiding the instability of adversarial objective functions.
### Experimental Validation
The authors demonstrate through experiments that TimeGCI can generate samples useful for prediction, and its performance reaches the level of existing benchmarks. This indicates that the method has significant advantages in generating high-quality time series data.
### Summary
By proposing a new generative framework, this paper effectively addresses key issues in time series data generation, particularly in maintaining conditional dynamics and joint distribution. This method is not only theoretically validated but also shows good performance in practical applications.