Abstract:This paper focuses on action recognition tasks oriented to the field of human–robot interaction, which is one of the major challenges in the robotic video understanding field. Previous approaches focus on designing temporal models, lack the ability to capture motion information and build contextual correlation models. This may result in robots being unable to effectively understand long-term video actions. To solve these two problems, this paper propose a novel video understanding framework including: an Adaptive Temporal Sensitivity and Motion Capture Network (ATSMC-Net) and a contextual scene reasoning module called Knowledge Function Graph Module (KFG-Module). The proposed ATSMC-Net can adaptively adjust the frame-level and pixel-level sensitive regions of temporal features to effectively capture motion information. To fuse contextual scene information for cross-temporal inference, the KFG-Module is introduced to achieve fine-grained video understanding based on the relationship between objects and actions. We evaluate the method using three public video understanding benchmarks, including Something-Something-V1&V2 and HMDB51. In addition, we present a dataset with real-world application scenarios of human–robot interactions to verify the effectiveness of our approach on mobile robots. The experimental results show that the proposed method can significantly improve the video understanding of the robots.

Survey on vision-based human action understanding

A Comprehensive Survey of Vision-Based Human Action Recognition Methods

View-invariant action recognition:a survey

Study of human action representation in video sequences

Action Recognition and Activity Understanding: A Review

A Survey of Human Action Recognition and Posture Prediction

Advances in Human Action Recognition: A Survey

A Survey on Human Action Recognition

A Survey on the Vision-Based Human Motion Recognition

A Survey of Video-based Action Quality Assessment

A survey on vision-based human action recognition

Human Behavior Analysis: A Survey on Action Recognition

Human Action Recognition Using Deep Learning Methods.

Human Action Recognition: A Taxonomy-Based Survey, Updates, and Opportunities

A Survey of Visual Analysis of Human Motion and Its Applications

Current Advances on Deep Learning-based Human Action Recognition from Videos: a Survey

Online human action detection and anticipation in videos: A survey

A Survey Of Activity Recognition And Understanding The Behavior In Video Survelliance

Video action recognition: A survey

A Review on Computer Vision-Based Methods for Human Action Recognition

Human–robot interaction-oriented video understanding of human actions