Abstract:In response to the difficulties in integrating multimodal data and insufficient model generalization ability in traditional cross-modal knowledge transfer, this article used the Transformer model to explore it in the new generation learning space. Firstly, the article analyzed the processing methods of data and models in cross-modal knowledge transfer, and explored the application of Transformer models in the learning space. This model used natural language processing to represent and extract textual features, Mel Frequency Cepstral Coefficients (MFCCs) to represent and extract audio features, and Faster R-CNN (Faster Region-based Convolutional Neural Network) to represent and extract image features. The article also discussed the implementation process of the Transformer model functionality. The experiment used data from four datasets, including Quora Question Pairs, to test the performance of the model's cross-modal knowledge transfer through intelligent question answering and task analysis. In single type data testing, the accuracy and recall of the model in this article were better than the comparison model in the three types of data. The highest accuracy and recall in the test set were 91% and 93%, respectively. In the most challenging multimodal intelligent question answering test, the speech-image question answering method achieved an accuracy rate of 89% in answering open questions, indicating that the model had good multimodal data fusion ability. In the analysis experiment of 6 homework prone knowledge points on images with text annotations, the induction accuracy reached 85%, indicating that the model had strong generalization ability. The experimental results showed that the Transformer model had good cross-modal knowledge transfer performance, providing a reference for subsequent research on cross-modal knowledge transfer in the new generation learning space.

MM-Transformer: A Transformer-Based Knowledge Graph Link Prediction Model That Fuses Multimodal Features

Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Multi-hop neighbor fusion enhanced hierarchical transformer for multi-modal knowledge graph completion

TGformer: A Graph Transformer Framework for Knowledge Graph Embedding

MLSFF: Multi-level structural features fusion for multi-modal knowledge graph completion

TE-TFN: A Text-enhanced Transformer Fusion Network for Multimodal Knowledge Graph Completion

Knowledge Graph Completion with Pre-trained Multimodal Transformer and Twins Negative Sampling

Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning

Kformer: Knowledge Injection in Transformer Feed-Forward Layers

Convolutional Models with Multi-Feature Fusion for Effective Link Prediction in Knowledge Graph Embedding

IMF: Interactive Multimodal Fusion Model for Link Prediction

MERGE: A Modal Equilibrium Relational Graph Framework for Multi-Modal Knowledge Graph Completion

The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework.

Utilization of transformer model in multimodal data fusion learning: Cross-modal knowledge transfer in the new generation learning space

Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos

TIVA-KG: A Multimodal Knowledge Graph with Text, Image, Video and Audio

Meta-Transformer: A Unified Framework for Multimodal Learning

Fusing Visual and Textual Content for Knowledge Graph Embedding Via Dual-Track Model

A Relation-Aware Heterogeneous Graph Transformer on Dynamic Fusion for Multimodal Classification Tasks

Multimodal Graph Transformer for Multimodal Question Answering