Loose Social-Interaction Recognition in Real-world Therapy Scenarios

Abid Ali,Rui Dai,Ashish Marisetty,Guillaume Astruc,Monique Thonnat,Jean-Marc Odobez,Susanne Thümmler,Francois Bremond

2024-09-30

Abstract:The computer vision community has explored dyadic interactions for atomic actions such as pushing, carrying-object, etc. However, with the advancement in deep learning models, there is a need to explore more complex dyadic situations such as loose interactions. These are interactions where two people perform certain atomic activities to complete a global action irrespective of temporal synchronisation and physical engagement, like cooking-together for example. Analysing these types of dyadic-interactions has several useful applications in the medical domain for social-skills development and mental health diagnosis. To achieve this, we propose a novel dual-path architecture to capture the loose interaction between two individuals. Our model learns global abstract features from each stream via a CNNs backbone and fuses them using a new Global-Layer-Attention module based on a cross-attention strategy. We evaluate our model on real-world autism diagnoses such as our Loose-Interaction dataset, and the publicly available Autism dataset for loose interactions. Our network achieves baseline results on the Loose-Interaction and SOTA results on the Autism datasets. Moreover, we study different social interactions by experimenting on a publicly available dataset i.e. NTU-RGB+D (interactive classes from both NTU-60 and NTU-120). We have found that different interactions require different network designs. We also compare a slightly different version of our method by incorporating time information to address tight interactions achieving SOTA results.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to identify loose social interactions in real - world treatment scenarios. Specifically, it focuses on the complex, asynchronous, and asymmetric interactive behaviors between two people, which do not require temporal synchronization or physical contact when completing a global task. For example, during the process of cooking together, one person may be the main operator (such as a chef), while the other person assists (such as preparing ingredients). This type of interaction has important application value in medical fields such as autism spectrum disorder (ASD) diagnosis, social skill development, and mental health assessment.

To solve this problem, the author proposes a new dual - path architecture to capture the loose interactions between two people. The model extracts the global abstract features of each stream through the 3D convolutional neural network (CNNs) backbone and uses a new Global - Layer - Attention module based on the cross - attention strategy for fusion. In addition, the author also verifies the effectiveness of this model on real - world autism diagnosis datasets and other public datasets, demonstrating its superior performance in the loose - interaction - identification task.

In summary, the core problem of the paper is **how to effectively identify and understand the complex, asynchronous, and asymmetric human interactive behaviors in the real world**, and an innovative deep - learning framework is proposed for this purpose.