Abstract:Knowledge based visual question-answering is an emerging technique that combines computer vision and natural language processing to address image-based questions. This approach requires the model to possess internal reasoning ability and incorporate external knowledge to enhance its generalization performance. Knowledge graphs are commonly employed to integrate various types of knowledge, such as image knowledge, text knowledge, and basic common sense, into visual language question answering models, significantly enhancing their interpretability. However, introducing knowledge into these models often involves retrieval and pre-training methods, which can introduce noise due to the local correlation among knowledge sources and the existence of modality gaps, thereby affecting the model's performance. To mitigate this issue, this paper proposes a multimodal knowledge extraction approach based on distributed representation learning. The approach models inexpressible multimodal facts using explicit triples, considering the semantic gap between visual and textual modalities. Cross-modal representations are obtained through an attention mechanism, resulting in knowledge triplets comprising visual features, cross-modal representations, and text features. By incorporating these knowledge triplets into the visual language question answering model, the task is transformed into pattern matching using knowledge triples. The proposed approach comprehensively considers multiple factors, is not restricted to specific forms of knowledge, and can effectively incorporate a substantial amount of knowledge. Experimental results demonstrate its superiority, with a performance improvement of 0.5% on the OKVQA dataset compared to the baseline model, highlighting its strong generalization ability.

Knowledge-integrated Multi-modal Movie Turning Point Identification

SKANet - Structured Knowledge-Aware Network for Visual Dialog.

Multi-modal Recommendation Based on Knowledge Graph

MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations

Knowledge-enhanced Multi-perspective Video Representation Learning for Scene Recognition

Knowledge Integration Networks for Action Recognition

Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI

Multimodal Knowledge Triple Extraction Based on Representation Learning

Generating Event-oriented Attribution for Movies via Two-Stage Prefix-Enhanced Multimodal LLM

Exploiting spatio‐temporal knowledge for video action recognition

Movie Question Answering via Textual Memory and Plot Graph

Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition

Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey

TIVA-KG: A Multimodal Knowledge Graph with Text, Image, Video and Audio

Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation

Research on the application of knowledge mapping and knowledge structure construction based on adaptive learning model

An Intention Multiple-Representation Model with Expanded Information

Video summarization via knowledge-aware multimodal deep networks

Learning a Contextual Multi-Thread Model for Movie/TV Scene Segmentation

Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning

Knowledge-aware Multimodal Dialogue Systems.