Abstract:Currently, existing action recognition methods mainly use a data-driven method to extract spatio-temporal representations of actions for recognition. However, this method may face performance bottlenecks. At the same time, existing action recognition methods are easily affected by the bias of scene information and object information in videos. In order to explore the essential causal relationship between factors and remove bias in action recognition, we introduce the theory of causal inference into the field of action recognition and propose a Knowledge-based Hierarchical Causal Inference Network (KHCIN) to help us step toward a new direction of inference in action recognition. First, we construct a Knowledge-based Hierarchical Causal Graph (KHCG) to structurally represent the scene, object and motion knowledge of a video. Then, in the model inference stage, we perform factual causal inference on a video on the constructed KHCG, and then deploy counterfactual inference on the Direct Content Hierarchy (DCH) and Indirect Interaction Hierarchy (IIH) in the KHCG. For DCH, we intervene in the model at the decision level to highlight bias errors in the model predictions. For the IIH, we focus on intervening in the feature modelling process. The biased interactions are revealed by interrupting the information communication in the feature space. By comparing the results of factual and counterfactual inference, we can easily expose the biased information in the original representations and eliminate them. Driven by counterfactual causal inference, our approach can significantly improve the performance of action recognition while improving model explainability. Extensive experiments demonstrate the effectiveness of this method. We hope that KHCIN can provide some new ideas for better introduction of causal inference theory in the action recognition community in the future.

Knowledge-Aware Causal Inference Network for Visual Dialog.

SKANet - Structured Knowledge-Aware Network for Visual Dialog.

Context-Aware Graph Inference with Knowledge Distillation for Visual Dialog

Counterfactual Visual Dialog: Robust Commonsense Knowledge Learning from Unbiased Training

KBGN: Knowledge-Bridge Graph Network for Adaptive Vision-Text Reasoning in Visual Dialogue

Iterative Context-Aware Graph Inference for Visual Dialog

Heterogeneous Knowledge Network for Visual Dialog

Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog

You should know more: Learning external knowledge for visual dialog

Distilling Knowledge in Causal Inference for Unbiased Visual Question Answering.

Multi-Granularity Semantic Collaborative Reasoning Network for Visual Dialog

Coupling Attention and Convolution for Heuristic Network in Visual Dialog.

KINet: Incorporating Relevant Facts into Knowledge-Grounded Dialog Generation

Reasoning Visual Dialogs With Structural And Partial Observations

Knowledge-Bridged Causal Interaction Network for Causal Emotion Entailment

Relation-Aware Multi-hop Reasoning forVisual Dialog

A Knowledge-Based Hierarchical Causal Inference Network for Video Action Recognition

Multi-aware Coreference Relation Network for Visual Dialog

Causal Reinforcement Learning for Knowledge Graph Reasoning

A Knowledge-Enhanced Inferential Network for Cross-Modality Multi-hop VQA

Hybrid Graph Reasoning with Dynamic Interaction for Visual Dialog