Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition

Shijian Deng,Erin E. Kosloski,Siddhi Patel,Zeke A. Barnett,Yiyang Nan,Alexander Kaplan,Sisira Aarukapalli,William T. Doan,Matthew Wang,Harsh Singh,Pamela R. Rollins,Yapeng Tian

2024-03-23

Abstract:In this article, we introduce a novel problem of audio-visual autism behavior recognition, which includes social behavior recognition, an essential aspect previously omitted in AI-assisted autism screening research. We define the task at hand as one that is audio-visual autism behavior recognition, which uses audio and visual cues, including any speech present in the audio, to recognize autism-related behaviors. To facilitate this new research direction, we collected an audio-visual autism spectrum dataset (AV-ASD), currently the largest video dataset for autism screening using a behavioral approach. It covers an extensive range of autism-associated behaviors, including those related to social communication and interaction. To pave the way for further research on this new problem, we intensively explored leveraging foundation models and multimodal large language models across different modalities. Our experiments on the AV-ASD dataset demonstrate that integrating audio, visual, and speech modalities significantly enhances the performance in autism behavior recognition. Additionally, we explored the use of a post-hoc to ad-hoc pipeline in a multimodal large language model to investigate its potential to augment the model's explanatory capability during autism behavior recognition. We will release our dataset, code, and pre-trained models.

Audio and Speech Processing,Artificial Intelligence,Computation and Language,Computer Vision and Pattern Recognition,Machine Learning,Multimedia

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use audio and visual information to identify autism - related behaviors, especially social behaviors, in autism spectrum disorder (ASD) screening. Traditional methods mainly rely on parent - reporting tools, which are suitable for children under 30 months old but are subjective and cannot cover a wider age range. In addition, most of the existing research focuses on the visual modality, ignoring the importance of audio information in identifying difficulties in social interaction. Therefore, this study aims to improve this situation by introducing a brand - new multimodal dataset (A V - ASD) and a multimodal behavior recognition task, in order to achieve more comprehensive and accurate autism behavior recognition.

Specifically, the main contributions of the paper include:
1. **Constructed a comprehensive autism behavior recognition dataset**: This dataset contains 928 video clips, covering 10 categories, and is currently the largest multimodal dataset for autism screening. These videos not only include repetitive stereotyped behaviors, but also pay special attention to social interaction behaviors related to autism diagnosis.
2. **Benchmarked different base models**: Researchers used a variety of advanced base models (such as CLIP, ImageBind and Whisper) for feature extraction and explored the performance under different modality combinations. It was found that specialized multimodal large - language models (MLLMs) can effectively identify autism behaviors by integrating audio and visual cues.
3. **Proposed a posterior - to - improvisation framework**: This framework aims to alleviate the catastrophic forgetting problem in the instruction - tuning process and at the same time improve the interpretability of model predictions.

Through these contributions, this study provides a new direction for autism behavior recognition, especially in multimodal data processing and model interpretability.