FLAASH: Flow-Attention Adaptive Semantic Hierarchical Fusion for Multi-Modal Tobacco Content Analysis

Naga VS Raviteja Chappa,Page Daniel Dobbs,Bhiksha Raj,Khoa Luu
2024-10-26
Abstract:The proliferation of tobacco-related content on social media platforms poses significant challenges for public health monitoring and intervention. This paper introduces a novel multi-modal deep learning framework named Flow-Attention Adaptive Semantic Hierarchical Fusion (FLAASH) designed to analyze tobacco-related video content comprehensively. FLAASH addresses the complexities of integrating visual and textual information in short-form videos by leveraging a hierarchical fusion mechanism inspired by flow network theory. Our approach incorporates three key innovations, including a flow-attention mechanism that captures nuanced interactions between visual and textual modalities, an adaptive weighting scheme that balances the contribution of different hierarchical levels, and a gating mechanism that selectively emphasizes relevant features. This multi-faceted approach enables FLAASH to effectively process and analyze diverse tobacco-related content, from product showcases to usage scenarios. We evaluate FLAASH on the Multimodal Tobacco Content Analysis Dataset (MTCAD), a large-scale collection of tobacco-related videos from popular social media platforms. Our results demonstrate significant improvements over existing methods, outperforming state-of-the-art approaches in classification accuracy, F1 score, and temporal consistency. The proposed method also shows strong generalization capabilities when tested on standard video question-answering datasets, surpassing current models. This work contributes to the intersection of public health and artificial intelligence, offering an effective tool for analyzing tobacco promotion in digital media.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The paper attempts to address the challenge of comprehensively analyzing tobacco-related video content on social media platforms. Specifically, the paper focuses on the following aspects: 1. **Modality Imbalance**: Existing methods often fail to achieve a balance when processing video and text features, leading to poor fusion effects and incomplete understanding of the content. 2. **Temporal Dynamics**: The temporal characteristics of tobacco use patterns and product displays in videos are often overlooked or insufficiently captured. 3. **Contextual Nuances**: Subtle contextual cues contained in visual and textual modalities are often ignored, resulting in inaccurate interpretation of tobacco-related content. 4. **Content Diversity**: The diversity of tobacco-related content, ranging from close-up product displays to wide-angle social scenes, poses significant challenges to the understanding and analysis by current models. To address these issues, the paper proposes a multimodal deep learning framework named Flow-Attention Adaptive Semantic Hierarchical Fusion (FLAASH). FLAASH tackles these challenges through the following key innovations: - **Flow-Attention Mechanism**: Captures subtle interactions between visual and textual modalities. - **Adaptive Weighting Scheme**: Dynamically adjusts the importance of different layers. - **Gating Mechanism**: Selectively emphasizes relevant features. Through these innovations, FLAASH can effectively process and analyze diverse tobacco-related content, from product displays to usage scenarios. The paper evaluates FLAASH on the Multimodal Tobacco Content Analysis Dataset (MTCAD), showing that it significantly outperforms existing methods in terms of classification accuracy, F1 score, and temporal consistency. Additionally, FLAASH also surpasses current models on standard video question-answering datasets, demonstrating its strong generalization capability. In summary, the paper aims to provide an effective tool through the FLAASH framework for analyzing tobacco promotion content on social media platforms, thereby supporting public health monitoring and intervention.