UTS Submission to ActivityNet Challenge 2017

Linchao Zhu,Yi Yang

2018-01-01

Abstract:We present our submission to the trimmed action recognition task in ActivityNet Challenge 2017. We mainly train the two-stream ConvNets and propose a multi-scale attention model to learn a compact video representation. We found that the multi-scale attention model outperforms average pooling on the Kinetics dataset.

What problem does this paper attempt to address?

CUHK ETHZ SIAT Submission to ActivityNet Challenge 2016

Yuanjun Xiong,Limin Wang,Zhe Wang,Bowen Zhang,Hang Song,Wei Li,Dahua Lin,Yu Qiao,Luc Van Gool,Xiaoou Tang

DOI: https://doi.org/10.48550/arxiv.1608.00797

2016-01-01

Abstract:This paper presents the method that underlies our submission to the untrimmed video classification task of ActivityNet Challenge 2016. We follow the basic pipeline of temporal segment networks and further raise the performance via a number of other techniques. Specifically, we use the latest deep model architecture, e.g., ResNet and Inception V3, and introduce new aggregation schemes (top-k and attention-weighted pooling). Additionally, we incorporate the audio as a complementary channel, extracting relevant information via a CNN applied to the spectrograms. With these techniques, we derive an ensemble of deep models, which, together, attains a high classification accuracy (mAP $93.23\%$) on the testing set and secured the first place in the challenge.
Trimmed Action Recognition, Dense-Captioning Events in Videos, and Spatio-temporal Action Localization with Focus on ActivityNet Challenge 2019

Zhaofan Qiu,Dong Li,Yehao Li,Qi Cai,Yingwei Pan,Ting Yao

DOI: https://doi.org/10.48550/arXiv.1906.07016

2019-06-14

Computer Vision and Pattern Recognition

Abstract:This notebook paper presents an overview and comparative analysis of our systems designed for the following three tasks in ActivityNet Challenge 2019: trimmed action recognition, dense-captioning events in videos, and spatio-temporal action localization.
Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge

Gurkirt Singh,Fabio Cuzzolin

DOI: https://doi.org/10.48550/arXiv.1607.01979

2016-07-13

Abstract:Current state-of-the-art human activity recognition is focused on the classification of temporally trimmed videos in which only one action occurs per frame. We propose a simple, yet effective, method for the temporal detection of activities in temporally untrimmed videos with the help of untrimmed classification. Firstly, our model predicts the top k labels for each untrimmed video by analysing global video-level features. Secondly, frame-level binary classification is combined with dynamic programming to generate the temporally trimmed activity proposals. Finally, each proposal is assigned a label based on the global label, and scored with the score of the temporal activity proposal and the global score. Ultimately, we show that untrimmed video classification models can be used as stepping stone for temporal detection.

Computer Vision and Pattern Recognition
CUHK&SIAT submission for THUMOS15 action recognition challenge

Limin Wang,Zhe Wang,Yuanjun Xiong,Yu Qiao

2015-01-01

Abstract:This paper presents the method of our submission for THUMOS15 action recognition challenge. We propose a new action recognition system by exploiting very deep twostream ConvNets and Fisher vector representation of iDT features. Specifically, we utilize those successful very deep architectures in images such as GoogLeNet and VGGNet to design the two-stream ConvNets. From our experiments, we see that deeper architectures obtain higher performance for spatial nets. However, for temporal net, deeper architectures could not yield better recognition accuracy. We analyze that the UCF101 dataset is relatively very small and it is very hard to train such deep networks on the current action datasets. Compared with traditional iDT features, our implemented two-stream ConvNets significantly outperform them. We further combine the recognition scores of both two-stream ConvNets and iDT features, and achieve 68% mAP value on the validation dataset of THUMOS15.
Uts-cmu at thumos 2015

Zhongwen Xu,Linchao Zhu,Yi Yang,Alexander G Hauptmann

2015-01-01

Abstract:This notebook paper describes our solution from UTSCMU team in the THUMOS 2015 action recognition challenge. Our system contains two major components, video representation generated by VLAD encoding from ConvNet features and multi-skip improved Dense Trajectories. In addition, we explore optical flow ConvNet and acoustic features such as MFCC and ASR in our system. We demonstrate that our complete system can achieve state-of-the-art performance in large-scale action recognition tasks.
CBR-Net: Cascade Boundary Refinement Network for Action Detection: Submission to ActivityNet Challenge 2020 (task 1)

Xiang Wang,Baiteng Ma,Zhiwu Qing,Yongpeng Sang,Changxin Gao,Shiwei Zhang,Nong Sang

2020-01-01

Abstract:In this report, we present our solution for the task of temporal action localization (detection) (task 1) in ActivityNet Challenge 2020. The purpose of this task is to temporally localize intervals where actions of interest occur and predict the action categories in a long untrimmed video. Our solution mainly includes three components: 1) feature encoding: we apply three kinds of backbones, including TSN [7], Slowfast[3] and I3d[1], which are both pretrained on Kinetics dataset[2]. Applying these models, we can extract snippet-level video representations; 2) proposal generation: we choose BMN [5] as our baseline, base on which we design a Cascade Boundary Refinement Network (CBR-Net) to conduct proposal detection. The CBR-Net mainly contains two modules: temporal feature encoding, which applies BiLSTM to encode long-term temporal information; CBR module, which targets to refine the proposal precision under different parameter settings; 3) action localization: In this stage, we combine the video-level classification results obtained by the fine tuning networks to predict the category of each proposal. Moreover, we also apply to different ensemble strategies to improve the performance of the designed solution, by which we achieve 42.788% on the testing set of ActivityNet v1.3 dataset in terms of mean Average Precision metrics.
Temporal Convolution Based Action Proposal: Submission to ActivityNet 2017

Tianwei Lin,Xu Zhao,Zheng Shou

DOI: https://doi.org/10.48550/arXiv.1707.06750

2017-07-21

Computer Vision and Pattern Recognition

Abstract:In this notebook paper, we describe our approach in the submission to the temporal action proposal (task 3) and temporal action localization (task 4) of ActivityNet Challenge hosted at CVPR 2017. Since the accuracy in action classification task is already very high (nearly 90% in ActivityNet dataset), we believe that the main bottleneck for temporal action localization is the quality of action proposals. Therefore, we mainly focus on the temporal action proposal task and propose a new proposal model based on temporal convolutional network. Our approach achieves the state-of-the-art performances on both temporal action proposal task and temporal action localization task.
Complementary Boundary Generator with Scale-Invariant Relation Modeling for Temporal Action Localization: Submission to ActivityNet Challenge 2020

Haisheng Su,Jinyuan Feng,Hao Shao,Zhenyu Jiang,Manyuan Zhang,Wei Wu,Yu Liu,Hongsheng Li,Junjie Yan

DOI: https://doi.org/10.48550/arXiv.2007.09883

2020-07-20

Computer Vision and Pattern Recognition

Abstract:This technical report presents an overview of our solution used in the submission to ActivityNet Challenge 2020 Task 1 (\textbf{temporal action localization/detection}). Temporal action localization requires to not only precisely locate the temporal boundaries of action instances, but also accurately classify the untrimmed videos into specific categories. In this paper, we decouple the temporal action localization task into two stages (i.e. proposal generation and classification) and enrich the proposal diversity through exhaustively exploring the influences of multiple components from different but complementary perspectives. Specifically, in order to generate high-quality proposals, we consider several factors including the video feature encoder, the proposal generator, the proposal-proposal relations, the scale imbalance, and ensemble strategy. Finally, in order to obtain accurate detections, we need to further train an optimal video classifier to recognize the generated proposals. Our proposed scheme achieves the state-of-the-art performance on the temporal action localization task with \textbf{42.26} average mAP on the challenge testing set.
An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Dong Tian,Zhe-Ming Lu,Xiao Chen,Long-Hua Ma

DOI: https://doi.org/10.1007/s11042-020-08611-4

IF: 2.577

2020-01-01

Multimedia Tools and Applications

Abstract:Action recognition plays a central role in intelligent surveillance system, game-control, human-computer interaction, and so on. In this work, we design a multi-task framework that improves the recent Spatial-Temporal Graph Convolutional Networks (ST-GCN) for skeleton-based action recognition by introducing the attention mechanism and co-occurrence feature learning. Specifically, we use an attentional branch to pay more attention to more discriminating features and aggregates co-occurrence features from all joints globally in another branch. Additionally, our multi-task framework exploits the inherent correlation between branches to further enhance the classification accuracy and convergence speed. Experiments have been carried out on NTURGB+D and Kinetics human action dataset. The results clearly show that the accuracy of the proposed multi-task framework are distinguishably higher than ST-GCN and other mainstream methods for 3D action recognition.
Technical Report for ActivityNet Challenge 2022 – Temporal Action Localization

Shimin Chen,Wei Li,Jianyang Gu,Chen,Yandong Guo

2024-01-01

Abstract:In the task of temporal action localization of ActivityNet-1.3 datasets, we propose to locate the temporal boundaries of each action and predict action class in untrimmed videos. We first apply VideoSwinTransformer as feature extractor to extract different features. Then we apply a unified network following Faster-TAD to simultaneously obtain proposals and semantic labels. Last, we ensemble the results of different temporal action detection models which complement each other. Faster-TAD simplifies the pipeline of TAD and gets remarkable performance, obtaining comparable results as those of multi-step approaches.
Three Branches: Detecting Actions With Richer Features

Jin Xia,Jiajun Tang,Cewu Lu

DOI: https://doi.org/10.48550/arXiv.1908.04519

2019-08-13

Computer Vision and Pattern Recognition

Abstract:We present our three branch solutions for International Challenge on Activity Recognition at CVPR2019. This model seeks to fuse richer information of global video clip, short human attention and long-term human activity into a unified model. We have participated in two tasks: Task A, the Kinetics challenge and Task B, spatio-temporal action localization challenge. For Kinetics, we achieve 21.59% error rate. For the AVA challenge, our final model obtains 32.49% mAP on the test sets, which outperforms all submissions to the AVA challenge at CVPR 2018 for more than 10% mAP. As the future work, we will introduce human activity knowledge, which is a new dataset including key information of human activity.
UTS submission to Google YouTube-8M Challenge 2017.

Linchao Zhu,Yanbin Liu,Yi Yang

2017-01-01

Abstract:In this paper, we present our solution to Google YouTube-8M Video Classification Challenge 2017. We leveraged both video-level and frame-level features in the submission. For video-level classification, we simply used a 200-mixture Mixture of Experts (MoE) layer, which achieves GAP 0.802 on the validation set with a single model. For frame-level classification, we utilized several variants of recurrent neural networks, sequence aggregation with attention mechanism and 1D convolutional models. We achieved GAP 0.8408 on the private testing set with the ensemble model. The source code of our models can be found in url{this https URL}.
Multi-head attention-based two-stream EfficientNet for action recognition

Aihua Zhou,Yujun Ma,Wanting Ji,Ming Zong,Pei Yang,Min Wu,Mingzhe Liu

DOI: https://doi.org/10.1007/s00530-022-00961-3

IF: 3.9

2022-06-25

Multimedia Systems

Abstract:Recent years have witnessed the popularity of using two-stream convolutional neural networks for action recognition. However, existing two-stream convolutional neural network-based action recognition approaches are incapable of distinguishing some roughly similar actions in videos such as sneezing and yawning. To solve this problem, we propose a Multi-head Attention-based Two-stream EfficientNet (MAT-EffNet) for action recognition, which can take advantage of the efficient feature extraction of EfficientNet. The proposed network consists of two streams (i.e., a spatial stream and a temporal stream), which first extract the spatial and temporal features from consecutive frames by using EfficientNet. Then, a multi-head attention mechanism is utilized on the two streams to capture the key action information from the extracted features. The final prediction is obtained via a late average fusion, which averages the softmax score of spatial and temporal streams. The proposed MAT-EffNet can focus on the key action information at different frames and compute the attention multiple times, in parallel, to distinguish similar actions. We test the proposed network on the UCF101, HMDB51 and Kinetics-400 datasets. Experimental results show that the MAT-EffNet outperforms other state-of-the-art approaches for action recognition.

computer science, information systems, theory & methods
Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019

Xiaohan Wang,Yu Wu,Linchao Zhu,Yi Yang

DOI: https://doi.org/10.48550/arXiv.1906.09383

2019-06-22

Abstract:In this report, we present the Baidu-UTS submission to the EPIC-Kitchens Action Recognition Challenge in CVPR 2019. This is the winning solution to this challenge. In this task, the goal is to predict verbs, nouns, and actions from the vocabulary for each video segment. The EPIC-Kitchens dataset contains various small objects, intense motion blur, and occlusions. It is challenging to locate and recognize the object that an actor interacts with. To address these problems, we utilize object detection features to guide the training of 3D Convolutional Neural Networks (CNN), which can significantly improve the accuracy of noun prediction. Specifically, we introduce a Gated Feature Aggregator module to learn from the clip feature and the object feature. This module can strengthen the interaction between the two kinds of activations and avoid gradient exploding. Experimental results demonstrate our approach outperforms other methods on both seen and unseen test set.

Computer Vision and Pattern Recognition
Action Machine: Rethinking Action Recognition in Trimmed Videos

Jiagang Zhu,Wei Zou,Liang Xu,Yiming Hu,Zheng Zhu,Manyu Chang,Junjie Huang,Guan Huang,Dalong Du

2018-01-01

Abstract:Existing methods in video action recognition mostly do not distinguish human body from the environment and easily overfit the scenes and objects. In this work, we present a conceptually simple, general and high-performance framework for action recognition in trimmed videos, aiming at person-centric modeling. The method, called Action Machine, takes as inputs the videos cropped by person bounding boxes. It extends the Inflated 3D ConvNet (I3D) by adding a branch for human pose estimation and a 2D CNN for pose-based action recognition, being fast to train and test. Action Machine can benefit from the multi-task training of action recognition and pose estimation, the fusion of predictions from RGB images and poses. On NTU RGB-D, Action Machine achieves the state-of-the-art performance with top-1 accuracies of 97.2 cross-subject respectively. Action Machine also achieves competitive performance on another three smaller action recognition datasets: Northwestern UCLA Multiview Action3D, MSR Daily Activity3D and UTD-MHAD. Code will be made available.
Efficient Action Detection in Untrimmed Videos via Multi-Task Learning

Yi Zhu,Shawn Newsam

DOI: https://doi.org/10.48550/arXiv.1612.07403

2017-04-05

Abstract:This paper studies the joint learning of action recognition and temporal localization in long, untrimmed videos. We employ a multi-task learning framework that performs the three highly related steps of action proposal, action recognition, and action localization refinement in parallel instead of the standard sequential pipeline that performs the steps in order. We develop a novel temporal actionness regression module that estimates what proportion of a clip contains action. We use it for temporal localization but it could have other applications like video retrieval, surveillance, summarization, etc. We also introduce random shear augmentation during training to simulate viewpoint change. We evaluate our framework on three popular video benchmarks. Results demonstrate that our joint model is efficient in terms of storage and computation in that we do not need to compute and cache dense trajectory features, and that it is several times faster than its sequential ConvNets counterpart. Yet, despite being more efficient, it outperforms state-of-the-art methods with respect to accuracy.

Computer Vision and Pattern Recognition,Multimedia
YH Technologies at ActivityNet Challenge 2018

Ting Yao,Xue Li

DOI: https://doi.org/10.48550/arXiv.1807.00686

2018-06-29

Computer Vision and Pattern Recognition

Abstract:This notebook paper presents an overview and comparative analysis of our systems designed for the following five tasks in ActivityNet Challenge 2018: temporal action proposals, temporal action localization, dense-captioning events in videos, trimmed action recognition, and spatio-temporal action localization.
Msr asia msm at activitynet challenge 2016

Zhaofan Qiu,Dong Li,Chuang Gan,Ting Yao,Tao Mei,Yong Rui

2016-01-01

Abstract:This notebook paper presents overview and comparative analysis of our system designed for untrimmed video classification task in ActivityNet Challenge 2016. We investigate and exploit multiple spatio-temporal clues, ie, frames, motion (optical flow), and short video clips, using 2D or 3D convolutional neural networks (CNNs). The mechanism of different quantization methods are studied as well. Furthermore, improved dense trajectory with fisher vector encoding on long video clips and MFCC audio features are utilized. All activities are classified by late fusing the predictions of one-versus-rest linear SVMs learnt on each clue. Finally, OCR is employed to refine the prediction scores.
A Joint Model for Action Localization and Classification in Untrimmed Video with Visual Attention

Weimian Li,Wenmin Wang,Xiongtao Chen,Jinzhuo Wang,Ge Li

DOI: https://doi.org/10.1109/icme.2017.8019335

2017-01-01

Abstract:In this paper, we introduce a joint model that learns to directly localize the temporal bounds of actions in untrimmed videos as well as precisely classify what actions occur. Most existing approaches tend to scan the whole video to generate action instances, which are really inefficient. Instead, inspired by human perception, our model is formulated based on a recurrent neural network to observe different locations within a video over time. And, it is capable of producing temporal localizations by only observing a fixed number of fragments, and the amount of computation it performs is independent of input video size. The decision policy for determining where to look next is learned by REINFORCE which is powerful in non-differentiable settings. In addition, different from relevant ways, our model runs localization and classification serially, and possesses a strategy for extracting appropriate features to classify. We evaluate our model on ActivityNet dataset, and it greatly outperforms the baseline. Moreover, compared with a recent approach, we show that our serial design can bring about 9% increase in detection performance.
Transferable Knowledge-Based Multi-Granularity Aggregation Network for Temporal Action Localization: Submission to ActivityNet Challenge 2021

Haisheng Su,Peiqin Zhuang,Yukun Li,Dongliang Wang,Weihao Gan,Wei Wu,Yu Qiao

DOI: https://doi.org/10.48550/arXiv.2107.12618

2021-07-27

Computer Vision and Pattern Recognition

Abstract:This technical report presents an overview of our solution used in the submission to 2021 HACS Temporal Action Localization Challenge on both Supervised Learning Track and Weakly-Supervised Learning Track. Temporal Action Localization (TAL) requires to not only precisely locate the temporal boundaries of action instances, but also accurately classify the untrimmed videos into specific categories. However, Weakly-Supervised TAL indicates locating the action instances using only video-level class labels. In this paper, to train a supervised temporal action localizer, we adopt Temporal Context Aggregation Network (TCANet) to generate high-quality action proposals through ``local and global" temporal context aggregation and complementary as well as progressive boundary refinement. As for the WSTAL, a novel framework is proposed to handle the poor quality of CAS generated by simple classification network, which can only focus on local discriminative parts, rather than locate the entire interval of target actions. Further inspired by the transfer learning method, we also adopt an additional module to transfer the knowledge from trimmed videos (HACS Clips dataset) to untrimmed videos (HACS Segments dataset), aiming at promoting the classification performance on untrimmed videos. Finally, we employ a boundary regression module embedded with Outer-Inner-Contrastive (OIC) loss to automatically predict the boundaries based on the enhanced CAS. Our proposed scheme achieves 39.91 and 29.78 average mAP on the challenge testing set of supervised and weakly-supervised temporal action localization track respectively.

UTS Submission to ActivityNet Challenge 2017

CUHK ETHZ SIAT Submission to ActivityNet Challenge 2016

Trimmed Action Recognition, Dense-Captioning Events in Videos, and Spatio-temporal Action Localization with Focus on ActivityNet Challenge 2019

Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge

CUHK&SIAT submission for THUMOS15 action recognition challenge

Uts-cmu at thumos 2015

CBR-Net: Cascade Boundary Refinement Network for Action Detection: Submission to ActivityNet Challenge 2020 (task 1)

Temporal Convolution Based Action Proposal: Submission to ActivityNet 2017

Complementary Boundary Generator with Scale-Invariant Relation Modeling for Temporal Action Localization: Submission to ActivityNet Challenge 2020

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Technical Report for ActivityNet Challenge 2022 – Temporal Action Localization

Three Branches: Detecting Actions With Richer Features

UTS submission to Google YouTube-8M Challenge 2017.

Multi-head attention-based two-stream EfficientNet for action recognition

Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019

Action Machine: Rethinking Action Recognition in Trimmed Videos

Efficient Action Detection in Untrimmed Videos via Multi-Task Learning

YH Technologies at ActivityNet Challenge 2018

Msr asia msm at activitynet challenge 2016

A Joint Model for Action Localization and Classification in Untrimmed Video with Visual Attention

Transferable Knowledge-Based Multi-Granularity Aggregation Network for Temporal Action Localization: Submission to ActivityNet Challenge 2021