Educational tool for hospital-based training in family medicine.

C. Gibson

2014-10-01

Abstract:Hospitalists are educators in many teaching institutions throughout North America, and they are a vital part of the medical training system. In Canada, this role is not yet clearly defined, as 90% of practising hospitalists are trained as family medicine specialists.[1][1] Once they begin working as

What problem does this paper attempt to address?

Learning Visual Context for Group Activity Recognition.

Hangjie Yuan,Dong Ni

DOI: https://doi.org/10.1609/aaai.v35i4.16437

2021-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Group activity recognition aims to recognize an overall activity in a multi-person scene. Previous methods strive to reason on individual features. However, they under-explore the person-specific contextual information, which is significant and informative in computer vision tasks. In this paper, we propose a new reasoning paradigm to incorporate global contextual information. Specifically, we propose two modules to bridge the gap between group activity and visual context. The first is Transformer based Context Encoding (TCE) module, which enhances individual representation by encoding global contextual information to individual features and refining the aggregated information. The second is Spatial-Temporal Bilinear Pooling (STBiP) module. It firstly further explores pairwise relationships for the context encoded individual representation, then generates semantic representations via gated message passing on a constructed spatial-temporal graph. On their basis, we further design a two-branch model that integrates the designed modules into a pipeline. Systematic experiments demonstrate each module's effectiveness on either branch. Visualizations indicate that visual contextual cues can be aggregated globally by TCE. Moreover, our method achieves state-of-the-art results on two widely used benchmarks using only RGB images as input and 2D backbones.
Social Adaptive Module for Weakly-supervised Group Activity Recognition

Rui Yan,Lingxi Xie,Jinhui Tang,Xiangbo Shu,Qi Tian

DOI: https://doi.org/10.48550/arXiv.2007.09470

2020-07-19

Abstract:This paper presents a new task named weakly-supervised group activity recognition (GAR) which differs from conventional GAR tasks in that only video-level labels are available, yet the important persons within each frame are not provided even in the training data. This eases us to collect and annotate a large-scale NBA dataset and thus raise new challenges to GAR. To mine useful information from weak supervision, we present a key insight that key instances are likely to be related to each other, and thus design a social adaptive module (SAM) to reason about key persons and frames from noisy data. Experiments show significant improvement on the NBA dataset as well as the popular volleyball dataset. In particular, our model trained on video-level annotation achieves comparable accuracy to prior algorithms which required strong labels.

Computer Vision and Pattern Recognition
A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

Huafeng Wang,Tao Xia,Hanlin Li,Xianfeng Gu,Weifeng Lv,Yuehai Wang

DOI: https://doi.org/10.3390/math9243226

IF: 2.4

2021-01-01

Mathematics

Abstract:A very challenging task for action recognition concerns how to effectively extract and utilize the temporal and spatial information of video (especially temporal information). To date, many researchers have proposed various spatial-temporal convolution structures. Despite their success, most models are limited in further performance especially on those datasets that are highly time-dependent due to their failure to identify the fusion relationship between the spatial and temporal features inside the convolution channel. In this paper, we proposed a lightweight and efficient spatial-temporal extractor, denoted as Channel-Wise Spatial-Temporal Aggregation block (CSTA block), which could be flexibly plugged in existing 2D CNNs (denoted by CSTANet). The CSTA Block utilizes two branches to model spatial-temporal information separately. In temporal branch, It is equipped with a Motion Attention Module (MA), which is used to enhance the motion regions in a given video. Then, we introduced a Spatial-Temporal Channel Attention (STCA) module, which could aggregate spatial-temporal features of each block channel-wisely in a self-adaptive and trainable way. The final experimental results demonstrate that the proposed CSTANet achieved the state-of-the-art results on EGTEA Gaze++ and Diving48 datasets, and obtained competitive results on Something-Something V1&V2 at the less computational cost.
Entropy Guided Attention Network for Weakly-Supervised Action Localization.

Yi Cheng,Ying Sun,Hehe Fan,Tao Zhuo,Joo-Hwee Lim,Mohan Kankanhalli

DOI: https://doi.org/10.1016/j.patcog.2022.108718

IF: 8

2022-01-01

Pattern Recognition

Abstract:One major challenge of Weakly-supervised Temporal Action Localization (WTAL) is to handle diverse backgrounds in videos. To model background frames, most existing methods treat them as an additional action class. However, because background frames usually do not share common semantics, squeezing all the different background frames into a single class hinders network optimization. Moreover, the network would be confused and tends to fail when tested on videos with unseen background frames. To address this problem, we propose an Entropy Guided Attention Network (EGA-Net) to treat background frames as out-of-domain samples. Specifically, we design a two-branch module, where a domain branch detects whether a frame is an action by learning a class-agnostic attention map, and an action branch recognizes the action category of the frame by learning a class-specific attention map. By aggregating the two attention maps to model the joint domain-class distribution of frames, our EGA-Net can handle varying backgrounds. To train the class-agnostic attention map with only the video-level class labels, we propose an Entropy Guided Loss (EGL), which employs entropy as the supervision signal to distinguish action and background. Moreover, we propose a Global Similarity Loss (GSL) to enhance the action-specific attention map via action class center. Extensive experiments on THUMOS14, ActivityNet1.2 and ActivityNet1.3 datasets demonstrate the effectiveness of our EGA-Net. (C) 2022 Elsevier Ltd. All rights reserved.
Dynamical Attention Hypergraph Convolutional Network for Group Activity Recognition

Xiaolin Zhu,Dongli Wang,Jianxun Li,Rui Su,Qin Wan,Yan Zhou

DOI: https://doi.org/10.1109/TNNLS.2024.3422265

2024-07-29

Abstract:Recently, group activity recognition (GAR) has drawn growing interests in video analysis and computer vision communities. The current models of GAR tasks are often impractical in that they suppose that all interactions between actors are pairwise, which only models and leverages part of the information in real entire interactions. Motivated by this, we design a distinct dynamical attention hypergraph convolutional network framework, referred to as DAHGCN, for precise GAR, modeling the entire interactions and capturing the high-order relationships among involved actors in a real-life scenario. Specifically, to learn complementary feature representations for fine-grained GAR, a multilevel feature descriptor (MLFD) module is proposed. Furthermore, for learning higher order interaction relationships, we construct a DAHGCN to accommodate complex group interactions, which can dynamically change the topology of the hypergraph and learn these key representations by virtue of the "similarity-based shared nearest-neighbor (SSNN) clustering" and "attention mechanisms" on hypergraph. Finally, a multiscale temporal convolution (MSTC) module is utilized to explore various long-range temporal dynamic correlations across different frames. In addition, comprehensive experiments on three commonly used GAR datasets clearly demonstrate that, when compared with the state-of-the-art methods, our proposed method can achieve the most optimal performance.
Detector-Free Weakly Supervised Group Activity Recognition

Dongkeun Kim,Jinsung Lee,Minsu Cho,Suha Kwak

DOI: https://doi.org/10.48550/arXiv.2204.02139

2022-04-05

Abstract:Group activity recognition is the task of understanding the activity conducted by a group of people as a whole in a multi-person video. Existing models for this task are often impractical in that they demand ground-truth bounding box labels of actors even in testing or rely on off-the-shelf object detectors. Motivated by this, we propose a novel model for group activity recognition that depends neither on bounding box labels nor on object detector. Our model based on Transformer localizes and encodes partial contexts of a group activity by leveraging the attention mechanism, and represents a video clip as a set of partial context embeddings. The embedding vectors are then aggregated to form a single group representation that reflects the entire context of an activity while capturing temporal evolution of each partial context. Our method achieves outstanding performance on two benchmarks, Volleyball and NBA datasets, surpassing not only the state of the art trained with the same level of supervision, but also some of existing models relying on stronger supervision.

Computer Vision and Pattern Recognition
Augmented skeleton sequences with hypergraph network for self-supervised group activity recognition

Guoquan Wang,Mengyuan Liu,Hong Liu,Peini Guo,Ti Wang,Jingwen Guo,Ruijia Fan

DOI: https://doi.org/10.2139/ssrn.4674256

IF: 8

2024-04-06

Pattern Recognition

Abstract:Contrastive learning has been widely applied to self-supervised skeleton-based single-person action recognition. However, directly employing single-person contrastive learning techniques for multi-person skeleton-based Group Activity Recognition (GAR) suffers from some challenges. Firstly, single-person data augmentation strategies struggle to capture complex collaborations between actors in multi-person scenarios, resulting in poor generalization. Secondly, real-world uncertainties in the number of people make single-person methods fail to capture changing high-order actor relations. Finally, single-person methods treat each actor with equal importance for recognition, struggling to distinguish imbalanced contributions between individual and group activities. To this end, the coarse-to-fine A ugmented H ypergraph Net work (AHNet) is proposed for effective self-supervised GAR. Specifically, we introduce multi-person augmentation strategies to enhance the generalization of the model under complex actor collaboration scenarios. Moreover, a knowledge-masked hypergraph network is employed to enhance the adaptability of the model to capture varied high-order actor relations. Finally, coarse-to-fine contrast among key actors is conducted to mitigate the imbalanced contributions between individual and group levels. Extensive experiments on multiple datasets demonstrate that our AHNet achieves substantial improvements over state-of-the-art methods with various backbone architectures. Our code is available at https://github.com/WGQ109/AHNet .

computer science, artificial intelligence,engineering, electrical & electronic
Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph

Zhengcen Li,Xinle Chang,Yueran Li,Jingyong Su

2024-07-28

Abstract:Group Activity Recognition aims to understand collective activities from videos. Existing solutions primarily rely on the RGB modality, which encounters challenges such as background variations, occlusions, motion blurs, and significant computational overhead. Meanwhile, current keypoint-based methods offer a lightweight and informative representation of human motions but necessitate accurate individual annotations and specialized interaction reasoning modules. To address these limitations, we design a panoramic graph that incorporates multi-person skeletons and objects to encapsulate group activity, offering an effective alternative to RGB video. This panoramic graph enables Graph Convolutional Network (GCN) to unify intra-person, inter-person, and person-object interactive modeling through spatial-temporal graph convolutions. In practice, we develop a novel pipeline that extracts skeleton coordinates using pose estimation and tracking algorithms and employ Multi-person Panoramic GCN (MP-GCN) to predict group activities. Extensive experiments on Volleyball and NBA datasets demonstrate that the MP-GCN achieves state-of-the-art performance in both accuracy and efficiency. Notably, our method outperforms RGB-based approaches by using only estimated 2D keypoints as input. Code is available at <a class="link-external link-https" href="https://github.com/mgiant/MP-GCN" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition
An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Dong Tian,Zhe-Ming Lu,Xiao Chen,Long-Hua Ma

DOI: https://doi.org/10.1007/s11042-020-08611-4

IF: 2.577

2020-01-01

Multimedia Tools and Applications

Abstract:Action recognition plays a central role in intelligent surveillance system, game-control, human-computer interaction, and so on. In this work, we design a multi-task framework that improves the recent Spatial-Temporal Graph Convolutional Networks (ST-GCN) for skeleton-based action recognition by introducing the attention mechanism and co-occurrence feature learning. Specifically, we use an attentional branch to pay more attention to more discriminating features and aggregates co-occurrence features from all joints globally in another branch. Additionally, our multi-task framework exploits the inherent correlation between branches to further enhance the classification accuracy and convergence speed. Experiments have been carried out on NTURGB+D and Kinetics human action dataset. The results clearly show that the accuracy of the proposed multi-task framework are distinguishably higher than ST-GCN and other mainstream methods for 3D action recognition.
DECOMPL: Decompositional Learning with Attention Pooling for Group Activity Recognition from a Single Volleyball Image

Berker Demirel,Huseyin Ozkan

DOI: https://doi.org/10.48550/arXiv.2303.06439

2023-03-11

Computer Vision and Pattern Recognition

Abstract:Group Activity Recognition (GAR) aims to detect the activity performed by multiple actors in a scene. Prior works model the spatio-temporal features based on the RGB, optical flow or keypoint data types. However, using both the temporality and these data types altogether increase the computational complexity significantly. Our hypothesis is that by only using the RGB data without temporality, the performance can be maintained with a negligible loss in accuracy. To that end, we propose a novel GAR technique for volleyball videos, DECOMPL, which consists of two complementary branches. In the visual branch, it extracts the features using attention pooling in a selective way. In the coordinate branch, it considers the current configuration of the actors and extracts the spatial information from the box coordinates. Moreover, we analyzed the Volleyball dataset that the recent literature is mostly based on, and realized that its labeling scheme degrades the group concept in the activities to the level of individual actors. We manually reannotated the dataset in a systematic manner for emphasizing the group concept. Experimental results on the Volleyball as well as Collective Activity (from another domain, i.e., not volleyball) datasets demonstrated the effectiveness of the proposed model DECOMPL, which delivered the best/second best GAR performance with the reannotations/original annotations among the comparable state-of-the-art techniques. Our code, results and new annotations will be made available through GitHub after the revision process.
Unveiling group activity recognition: Leveraging Local–Global Context-Aware Graph Reasoning for enhanced actor–scene interactions

Xue Jiang,Linbo Qing,Jianglan Huang,Li Guo,Yonghong Peng

DOI: https://doi.org/10.1016/j.engappai.2024.108412

IF: 8

2024-04-14

Engineering Applications of Artificial Intelligence

Abstract:Group activity recognition aims to recognize holistic activity in multi-person scene, necessitating consideration of the interactions between actors and their surroundings. It has various applications, such as public surveillance, video analysis. Nonetheless, existing works merely extract scene features as a supplementary component of activity features, failing to adequately explore the interplay between the scene and actors. To address this limitation, this paper proposes a Local–Global Context-Aware Graph Reasoning Model (LG-CAGR), which leverages and reasons through local and global context features to gain deeper insights into group activity within the scene. In particular, we present an innovative feature extraction strategy to harness local location features and global scene attributes, effectively complementing actor features by capturing spatial group topology and determining relative positions between actors. Subsequently, we delve into these features by devising a local–global group reasoning module that deduces pair-wise interactions between actors and scenes within Graph Convolutional Network, comprehensively elucidating correlations between overall scene and local individuals to construct group-level features. Multi-graphs are constructed considering actor's features, scene features as nodes, and interactions as edges. A self-attention graph pooling network is introduced to automatically integrate key actor features and form rich group-level features to recognize group activity. The results on Collective Activity Dataset, Collective Activity Extended Dataset, Volleyball Dataset and Public Life in Public Space dataset have reached 94.0%, 97.7%, 92.7% and 56.1%. Compared with existing methods using the same backbone, we exceeded 1%, 2.1%, 0.3%, and 14.9% respectively, affirming the superiority of the proposed method compared with state-of-the-art methods.

automation & control systems,computer science, artificial intelligence,engineering, electrical & electronic, multidisciplinary
Flow-Assisted Motion Learning Network for Weakly-Supervised Group Activity Recognition

Muhammad Adi Nugroho,Sangmin Woo,Sumin Lee,Jinyoung Park,Yooseung Wang,Donguk Kim,Changick Kim

2024-05-28

Abstract:Weakly-Supervised Group Activity Recognition (WSGAR) aims to understand the activity performed together by a group of individuals with the video-level label and without actor-level labels. We propose Flow-Assisted Motion Learning Network (Flaming-Net) for WSGAR, which consists of the motion-aware actor encoder to extract actor features and the two-pathways relation module to infer the interaction among actors and their activity. Flaming-Net leverages an additional optical flow modality in the training stage to enhance its motion awareness when finding locally active actors. The first pathway of the relation module, the actor-centric path, initially captures the temporal dynamics of individual actors and then constructs inter-actor relationships. In parallel, the group-centric path starts by building spatial connections between actors within the same timeframe and then captures simultaneous spatio-temporal dynamics among them. We demonstrate that Flaming-Net achieves new state-of-the-art WSGAR results on two benchmarks, including a 2.8%p higher MPCA score on the NBA dataset. Importantly, we use the optical flow modality only for training and not for inference.

Computer Vision and Pattern Recognition,Image and Video Processing
Collaborative Activity Recognition with Heterogeneous Activity Sets and Privacy Preferences

Gabriele Civitarese,Juan Ye,Matteo Zampatti,Claudio Bettini

DOI: https://doi.org/10.3233/ais-210018

2021-01-01

Journal of Ambient Intelligence and Smart Environments

Abstract:One of the major challenges in Human Activity Recognition (HAR) based on machine learning is the scarcity of labeled data. Indeed, collecting a sufficient amount of training data to build a reliable recognition problem is often prohibitive. Among the many solutions in the literature to mitigate this issue, collaborative learning is emerging as a promising direction to distribute the annotation burden over multiple users that cooperate to build a shared recognition model. One of the major issues of existing methods is that they assume a static activity model with a fixed set of target activities. In this paper, we propose a novel approach that is based on Growing When Required (GWR) neural networks. A GWR network continuously adapts itself according to the input training data, and hence it is particularly suited when the users share heterogeneous sets of activities. Like in federated learning, for the sake of privacy preservation, each user contributes to the global activity classifier by sharing personal model parameters, and not by directly sharing data. In order to further mitigate privacy threats, we implement a strategy to avoid releasing model parameters that may indirectly reveal information about activities that the user specifically marked as private. Our results on two well-known publicly available datasets show the effectiveness and the flexibility of our approach.
HiGCIN: Hierarchical Graph-Based Cross Inference Network for Group Activity Recognition

Rui Yan,Lingxi Xie,Jinhui Tang,Xiangbo Shu,Qi Tian

DOI: https://doi.org/10.1109/tpami.2020.3034233

IF: 23.6

2023-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Group activity recognition (GAR) is a challenging task aimed at recognizing the behavior of a group of people. It is a complex inference process in which visual cues collected from individuals are integrated into the final prediction, being aware of the interaction between them. This paper goes one step further beyond the existing approaches by designing a Hierarchical Graph-based Cross Inference Network (HiGCIN), in which three levels of information, i.e., the body-region level, person level, and group-activity level, are constructed, learned, and inferred in an end-to-end manner. Primarily, we present a generic Cross Inference Block (CIB), which is able to concurrently capture the latent spatiotemporal dependencies among body regions and persons. Based on the CIB, two modules are designed to extract and refine features for group activities at each level. Experiments on two popular benchmarks verify the effectiveness of our approach, particularly in the ability to infer with multilevel visual cues. In addition, training our approach does not require individual action labels to be provided, which greatly reduces the amount of labor required in data annotation.
SoGAR: Self-supervised Spatiotemporal Attention-based Social Group Activity Recognition

Naga VS Raviteja Chappa,Pha Nguyen,Alexander H Nelson,Han-Seok Seo,Xin Li,Page Daniel Dobbs,Khoa Luu

2024-11-19

Abstract:This paper introduces a novel approach to Social Group Activity Recognition (SoGAR) using Self-supervised Transformers network that can effectively utilize unlabeled video data. To extract spatio-temporal information, we created local and global views with varying frame rates. Our self-supervised objective ensures that features extracted from contrasting views of the same video were consistent across spatio-temporal domains. Our proposed approach is efficient in using transformer-based encoders to alleviate the weakly supervised setting of group activity recognition. By leveraging the benefits of transformer models, our approach can model long-term relationships along spatio-temporal dimensions. Our proposed SoGAR method achieved state-of-the-art results on three group activity recognition benchmarks, namely JRDB-PAR, NBA, and Volleyball datasets, surpassing the current numbers in terms of F1-score, MCA, and MPCA metrics.

Computer Vision and Pattern Recognition
GCWNet: A Global Context-Weaving Network for Object Detection in Remote Sensing Images

Yulin Wu,Ke Zhang,Jingyu Wang,Yezi Wang,Qi Wang,Xuelong Li

DOI: https://doi.org/10.1109/tgrs.2022.3155899

IF: 8.2

2022-01-01

IEEE Transactions on Geoscience and Remote Sensing

Abstract:With practical applications such as environment surveillance, agricultural production, and disaster assessment, accurate object detection in remote sensing images is in high demand. Precise detection of object instances in remote sensing images remains considerably challenging due to dense instance stacking, large-scale variations, and complex backgrounds. To solve the mentioned issues, a novel global context-weaving network (GCWNet) is developed for object detection in remote sensing images. We propose two novel modules for feature extraction and refinement, which include the global context aggregation module (GCAM) and the feature refinement module (FRM). GCAM assembles a global context with high-level and low-level features through feature weaving, which facilitates dense object detection. Meanwhile, FRM convolves multiple receptive fields by combining different branches, thereby further refining the features and improving the feature distinction at different scales. Furthermore, we design to alleviate the sample imbalanced problem during training using focal loss and balanced L1 loss to improve object classification and regression, respectively. The experimental results indicate that GCWNet achieves superior performance in object classification and localization on the DOTA-v1.5 dataset, which illustrates the superiority of GCWNet.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics
Multi-dimensional convolution transformer for group activity recognition

Liu, Jinfu,Zhang, Zixin

DOI: https://doi.org/10.1007/s11042-024-19973-4

IF: 2.577

2024-09-17

Multimedia Tools and Applications

Abstract:Group activity recognition, which aims to understand the activity performed by a group of people, has attracted growing attention in the realm of computer vision over the past decade. In this paper, we propose a novel multi-dimensional convolution Transformer network for group activity recognition, which not only models spatial-temporal feature representations, but also combines channel information to analyze the spatial-temporal dependencies of individual actors. Specifically, we first construct a multi-scale feature extraction module in the feature extraction stage, which can exploit discriminative high-level and low-level feature representations. The multi-branching strategy combined with the dilated convolution can further capture multi-scale feature information in complex group scenarios. Then, to construct the inter-dependence among involved actors from different dimensions, we design a multi-dimensional convolution Transformer in the relational reasoning stage, which consists of the following three parts: a channel attention module, a spatial-temporal convolutional Transformer, and a spatial-temporal attention module. Finally, the final activity recognition result is obtained by using a softmax classifier. Extensive experiments on two public GAR datasets demonstrate that the recognition accuracy on the Volleyball Dataset and Collective Activity Dataset can reach 92.8% and 96.1%, respectively, which is a significant improvement compared with the mainstream methods in recent years.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
CERN: Confidence-Energy Recurrent Network for Group Activity Recognition

Tianmin Shu,Sinisa Todorovic,Song-Chun Zhu

DOI: https://doi.org/10.1109/cvpr.2017.453

2017-01-01

Abstract:This work is about recognizing human activities occurring in videos at distinct semantic levels, including individual actions, interactions, and group activities. The recognition is realized using a two-level hierarchy of Long ShortTerm Memory (LSTM) networks, forming a feed-forward deep architecture, which can be trained end-to-end. In comparison with existing architectures of LSTMs, we make two key contributions giving the name to our approach as Confidence-Energy Recurrent Network - CERN. First, instead of using the common softmax layer for prediction, we specify a novel energy layer (EL) for estimating the energy of our predictions. Second, rather than finding the common minimum-energy class assignment, which may be numerically unstable under uncertainty, we specify that the EL additionally computes the p-values of the solutions, and in this way estimates the most confident energy minimum. The evaluation on the Collective Activity and Volleyball datasets demonstrates: (i) advantages of our two contributions relative to the common softmax and energy-minimization formulations and (ii) a superior performance relative to the state-of-the-art approaches.
Group Activity Recognition Based on Temporal Semantic Sub-Graph Network.

DongLi Wang,Jia Liu,Yan Zhou

DOI: https://doi.org/10.1145/3529836.3529899

2022-01-01

Abstract:Group Activity Recognition is a very important and challenging task in the field of computer vision. Most of the proposed methods only extract the semantic or temporal information of video respectively, while ignoring the important relationship between temporal information and semantic information. In this paper, a more flexible and effective Spatial-Temporal Sub-Graph Network was proposed, which regards the features of each video frame thand e relationship between frames as nodes and edges. respectively. It uses Mixed Pooling Module (MPM)to pool and modify the basic features of video frames. Frame Feature Extraction Module (FFEM) learns node features by integrating context and updating relationship edges frequently, and the Frame Relationship Graph Module (FRGM) localizes each relationship sub-graph and maps each sub-graph into Euclidean space. In order to evaluate the performance of the Network, experiments on two public datasets in group activity recognition field have been conducted.
GLNet: Global Local Network for Weakly Supervised Action Localization

Shiwei Zhang,Lin Song,Changxin Gao,Nong Sang

DOI: https://doi.org/10.1109/tmm.2019.2959425

IF: 7.3

2020-10-01

IEEE Transactions on Multimedia

Abstract:In this paper, we address the challenging problem of weakly supervised spatio-temporal action localization for which only video-level action labels are available during training. To solve this problem, we propose an end-to-end Global Local Network (GLNet) to predict the probability distribution simultaneously in both spatial and temporal space. The proposed GLNet model includes two key components: a local spatial module and a global temporal module. The local spatial module aims to predict the frame-level spatial distribution by encoding short-term temporal information. In particular, we propose a Region Actionness Network (RAN) to select the target region boxes from the precomputed exhaustive proposals. The global temporal module can predict temporal distribution by a long-term temporal structure modelling. Specifically, we design a temporal fusion-and-excitation architecture on the top of several clips, and trained by a sparse loss function. Therefore, the proposed GLNet model can perform spatio-temporal action localization in an end-to-end manner. We evaluate the performance of GLNet on the J-HMDB and UCF101-24 datasets. The experimental results demonstrate GLNet achieves a significant margin against other state-of-the-art weakly supervised methods and even some fully supervised methods in terms of frame mean Average Precision (mAP) and the video mAP (called frame-mAP and video-mAP, respectively).

computer science, information systems,telecommunications, software engineering

Educational tool for hospital-based training in family medicine.

Learning Visual Context for Group Activity Recognition.

Social Adaptive Module for Weakly-supervised Group Activity Recognition

A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

Entropy Guided Attention Network for Weakly-Supervised Action Localization.

Dynamical Attention Hypergraph Convolutional Network for Group Activity Recognition

Detector-Free Weakly Supervised Group Activity Recognition

Augmented skeleton sequences with hypergraph network for self-supervised group activity recognition

Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

DECOMPL: Decompositional Learning with Attention Pooling for Group Activity Recognition from a Single Volleyball Image

Unveiling group activity recognition: Leveraging Local–Global Context-Aware Graph Reasoning for enhanced actor–scene interactions

Flow-Assisted Motion Learning Network for Weakly-Supervised Group Activity Recognition

Collaborative Activity Recognition with Heterogeneous Activity Sets and Privacy Preferences

HiGCIN: Hierarchical Graph-Based Cross Inference Network for Group Activity Recognition

SoGAR: Self-supervised Spatiotemporal Attention-based Social Group Activity Recognition

GCWNet: A Global Context-Weaving Network for Object Detection in Remote Sensing Images

Multi-dimensional convolution transformer for group activity recognition

CERN: Confidence-Energy Recurrent Network for Group Activity Recognition

Group Activity Recognition Based on Temporal Semantic Sub-Graph Network.

GLNet: Global Local Network for Weakly Supervised Action Localization