Multimodal Guidance Network for Missing-Modality Inference in Content Moderation

Zhuokai Zhao,Harish Palani,Tianyi Liu,Lena Evans,Ruth Toner

2024-08-02

Abstract:Multimodal deep learning, especially vision-language models, have gained significant traction in recent years, greatly improving performance on many downstream tasks, including content moderation and violence detection. However, standard multimodal approaches often assume consistent modalities between training and inference, limiting applications in many real-world use cases, as some modalities may not be available during inference. While existing research mitigates this problem through reconstructing the missing modalities, they unavoidably increase unnecessary computational cost, which could be just as critical, especially for large, deployed infrastructures in industry. To this end, we propose a novel guidance network that promotes knowledge sharing during training, taking advantage of the multimodal representations to train better single-modality models to be used for inference. Real-world experiments in violence detection shows that our proposed framework trains single-modality models that significantly outperform traditionally trained counterparts, while avoiding increases in computational cost for inference.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

This paper aims to address the issue of missing modality inference in multimodal deep learning, particularly in the applications of content moderation and violence detection tasks. Traditional multimodal approaches typically assume modality consistency during both training and inference phases. However, in real-world applications, certain modalities may be unavailable during the inference stage, which limits the applicability of multimodal models. Some existing studies mitigate this issue by reconstructing missing modality features, but this approach adds unnecessary computational costs, especially in large-scale industrial deployments.

To solve this problem, the paper proposes a novel guiding network framework that promotes knowledge sharing during training, utilizing multimodal representations to train better unimodal models for inference. Experimental results show that the unimodal models trained with this framework significantly outperform those trained with traditional methods in violence detection tasks, and avoid the increased computational costs during inference. This new approach not only improves model performance but also reduces latency and costs, making it more suitable for real-time content moderation and violence detection applications.