David W. Romero,Erik J. Bekkers,Jakub M. Tomczak,Mark Hoogendoorn
Abstract:Leveraging the symmetries inherent to specific data domains for the construction of equivariant neural networks has lead to remarkable improvements in terms of data efficiency and generalization. However, most existing research focuses on symmetries arising from planar and volumetric data, leaving a crucial data source largely underexplored: time-series. In this work, we fill this gap by leveraging the symmetries inherent to time-series for the construction of equivariant neural network. We identify two core symmetries: *scale and translation*, and construct scale-translation equivariant neural networks for time-series learning. Intriguingly, we find that scale-translation equivariant mappings share strong resemblance with the wavelet transform. Inspired by this resemblance, we term our networks Wavelet Networks, and show that they perform nested non-linear wavelet-like time-frequency transforms. Empirical results show that Wavelet Networks outperform conventional CNNs on raw waveforms, and match strongly engineered spectrogram techniques across several tasks and time-series types, including audio, environmental sounds, and electrical signals. Our code is publicly available at <a class="link-external link-https" href="https://github.com/dwromero/wavelet_networks" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
### Problems the Paper Attempts to Solve
This paper aims to address a key issue in time series data processing, namely how to leverage the inherent symmetries of time series (especially scale and translation symmetries) to construct equivariant neural networks. Specifically, the paper proposes the following points:
1. **Identifying Core Symmetries in Time Series**:
- The authors identify two fundamental symmetries in time series: scale symmetry and translation symmetry.
- These symmetries explain various phenomena observed in time series, such as time translation, phase shift, time scaling, resolution change, pitch change, seasonal events, etc.
2. **Constructing Scale-Translation Equivariant Neural Networks**:
- By utilizing scale-translation group equivariant convolutions, the authors construct a new neural network architecture called Wavelet Networks.
- When the input signal undergoes translation, scaling, or a combination of both, all intermediate layers transform in the same way, thereby maintaining symmetry consistency.
3. **Comparison with Existing Methods**:
- Experimental results show that Wavelet Networks significantly outperform traditional CNNs in processing raw waveforms and are comparable to highly engineered spectrogram techniques (such as Mel spectrograms) across multiple tasks and different types of time series data (e.g., audio, environmental sounds, electrical signals).
4. **Theoretical Foundation**:
- The paper provides a detailed introduction to the concepts of group equivariance and group invariance and explains how these concepts can be applied to time series data.
- The authors also discuss why traditional 2D CNNs may not be suitable for effective time series learning when dealing with spectro-temporal representations.
### Summary
The main contribution of this paper is filling the gap in the research of equivariant neural networks for time series data processing. By constructing scale-translation equivariant Wavelet Networks, the paper enhances data efficiency and generalization capability. This approach is not only theoretically significant but also performs well in practical applications.