Abstract:Human beings always engage in a vast range of activities and tasks that demonstrate their ability to adapt to different scenarios. These activities can range from the simplest daily routines, like walking and sitting, to multi-level complex endeavors such as cooking a four-course meal. Any human activity can be represented as a temporal sequence of actions performed to achieve a certain goal. Unlike the time series datasets extracted from electronics or machines, these action sequences are highly disparate in their nature – the time to finish a sequence of actions can vary between different persons. Therefore, understanding the dynamics of these sequences is essential for many downstream tasks such as activity length prediction, goal prediction, next-action recommendation, etc. Existing neural network-based approaches that learn a continuous-time activity sequence (or CTAS) are limited to the presence of only visual data or are designed specifically for a particular task, i.e. , limited to next action or goal prediction. In this paper, we present ProActive , a neural marked temporal point process (MTPP) framework for modeling the continuous-time distribution of actions in an activity sequence while simultaneously addressing three high-impact problems – next action prediction, sequence-goal prediction, and end-to-end sequence generation. Specifically, we utilize a self-attention module with temporal normalizing flows to model the influence and the inter-arrival times between actions in a sequence. Moreover, for time-sensitive prediction, we perform an early detection of sequence goal via a constrained margin-based optimization procedure. This in-turn allows ProActive to predict the sequence goal using a limited number of actions. In addition, we propose a novel addition over the ProActive model, called ProActive++ , that can handle variations in the order of actions, i.e. , different methods of achieving a given goal. We demonstrate that this variant can learn the order in which the person or actor prefers to do their actions. Extensive experiments on sequences derived from three activity recognition datasets show the significant accuracy boost of our ProActive and ProActive++ over the state-of-the-art in terms of action and goal prediction, and the first-ever application of end-to-end action sequence generation.

Spatio-Temporal Activity Recognition for Evolutionary Search Behavior Prediction

Learning Visual Context for Group Activity Recognition.

Human Activity Recognition based on Dynamic Spatio-Temporal Relations

Deep Behavior Tracing with Multi-level Temporality Preserved Embedding

Human Activity Recognition Based on Deep-Temporal Learning Using Convolution Neural Networks Features and Bidirectional Gated Recurrent Unit With Features Selection

TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition

Video Action Recognition Via Neural Architecture Searching

Time Perception Machine: Temporal Point Processes for the When, Where and What of Activity Prediction

Tapestry of Time and Actions: Modeling Human Activity Sequences using Temporal Point Process Flows

Long Term Activity Prediction in First Person Viewpoint

EA-LSTM: Evolutionary attention-based LSTM for time series prediction

Deep Generative Domain Adaptation with Temporal Relation Knowledge for Cross-User Activity Recognition

Progressive Relation Learning for Group Activity Recognition

Learning to Describe Collective Search Behavior of Evolutionary Algorithms in Solution Space.

Deep Dilation on Multimodality Time Series for Human Activity Recognition.

Spatio-temporal learning with the online finite and infinite echo-state Gaussian processes

Learning Generative State Space Models for Active Inference

3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks

Monte Carlo Tree Search for Scheduling Activity Recognition

Evolutionary Design of Convolutional Neural Networks for Human Activity Recognition in Sensor-Rich Environments