Abstract:<p>Video action anticipation aims to predict future action categories from observed frames. Current state-of-the-art approaches mainly resort to recurrent neural networks to encode history information into hidden states, and predict future actions from the hidden representations. It is well known that the recurrent pipeline is inefficient in capturing long-term information which may limit its performance in predication task. To address this problem, this paper proposes a simple yet efficient Temporal Transformer with Progressive Prediction (TTPP) framework, which repurposes a Transformer-style architecture to aggregate observed features, and then leverages a light-weight network to progressively predict future features and actions. Specifically, predicted features along with predicted probabilities are accumulated into the inputs of subsequent prediction. We evaluate our approach on three action datasets, namely TVSeries, THUMOS-14, and TV-Human-Interaction. Additionally we also conduct a comprehensive study for several popular aggregation and prediction strategies. Extensive results show that TTPP not only outperforms the state-of-the-art methods but also more efficient.</p>
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to efficiently perform future action prediction (action anticipation) in video understanding. Specifically, existing methods mainly rely on recurrent neural networks (such as LSTM) to encode historical information and predict future actions from it. However, these recurrent models are less efficient in capturing long - term information, which limits their performance in prediction tasks. To overcome this problem, this paper proposes a simple and efficient framework - Temporal Transformer with Progressive Prediction (TTPP). This framework re - utilizes the Transformer architecture to aggregate observed features and gradually predicts future features and actions through a lightweight network.
### Main Contributions
1. **Propose the TTPP Framework**: This framework uses a Transformer - style architecture to aggregate information and a lightweight module to predict future actions.
2. **Design the Progressive Prediction Module**: This module can gradually predict future features and actions and achieves the best performance on multiple datasets.
3. **Comprehensively Study Multiple Aggregation and Prediction Strategies**: Including methods such as temporal convolution, LSTM, and single - time prediction, verifying the superiority of TTPP.
### Method Overview
1. **Problem Definition**: Given a video \( V=\{I_1, I_2,\ldots, I_L\} \) containing \( L \) frames, the task is to predict the action labels \( y_{t + 1}^L=\{y_{t+1}, y_{t + 2},\ldots, y_L\} \) from the \( (t + 1)\) - th frame to the \( L\) - th frame from the first \( t \) frames \( V_t^1=\{I_1, I_2,\ldots, I_t\} \).
2. **Overall Framework**:
- **Video Pre - processing**: Divide a long video into multiple non - overlapping segments, each containing the same number of consecutive frames.
- **Feature Extraction**: Use the network \( g_{\text{enc}} \) to map each video segment to a representation \( f_t = g_{\text{enc}}(I_t') \).
- **Temporal Transformer Module (TTM)**: Aggregate the representations of \( t \) consecutive segments \( S_t = g_{\text{ttm}}(f_1, f_2,\ldots, f_t) \).
- **Progressive Prediction Module (PPM)**: Gradually predict future features and actions.
3. **Temporal Transformer Module (TTM)**:
- **Self - Attention Mechanism**: Use the current frame feature \( f_t \) as the query, and the historical frame features \( [f_1, f_2,\ldots, f_t] \) as the memory (key and value).
- **Multi - Head Attention Mechanism**: Further improve the ability to model complex action videos.
4. **Progressive Prediction Module (PPM)**:
- **Initial Prediction Block**: Input the aggregated historical representation \( S_t \), and predict the features and action probabilities of the next segment.
- **Progressive Prediction Blocks with Shared Parameters**: Accumulate previous prediction results and \( S_t \), and further predict subsequent features and actions.
### Experimental Results
1. **Datasets**: Evaluated on three datasets, namely TVSeries, THUMOS - 14, and TV - Human - Interaction.
2. **Evaluation Metrics**:
- TVSeries: Use per - frame calibrated average precision (cAP).
- THUMOS - 14: Use per - frame average precision (mAP).
- TV - Human - Interaction: Use classification accuracy (ACC).
3. **Experimental Results**:
- Using two - stream features, TTPP reaches 77.9% (mean cAP) on TVSeries, 40.9% (mAP) on THUMOS - 14, and on TV - Human - Interaction.