Abstract:Objective: The Chinese description of images combines the two directions of computer vision and natural language processing. It is a typical representative of multi-mode and cross-domain problems with artificial intelligence algorithms. The image Chinese description model needs to output a Chinese description for each given test picture, describe the sentence requirements to conform to the natural language habits, and point out the important information in the image, covering the main characters, scenes, actions and other content. Since the current open source datasets are mostly in English, the research on the direction of image description is mainly in English. Chinese descriptions usually have greater flexibility in syntax and lexicalization, and the challenges of algorithm implementation are also large. Therefore, only a few people have studied image descriptions, especially Chinese descriptions. Methods: This study attempts to derive a model of image description generation from the Flickr8k-cn and Flickr30k-cn datasets. At each time period of the description, the model can decide whether to rely more on images or text information. The model captures more important information from the image to improve the richness and accuracy of the Chinese description of the image. The image description data set of this study is mainly composed of Chinese description sentences. The method consists of an encoder and a decoder. The encoder is based on a convolutional neura l network. The decoder is based on a long-short memory network and is composed of a multi-modal summary generation network. Results: Experiments on Flickr8k-cn and Flickr30k-cn Chinese datasets show that the proposed method is superior to the existing Chinese abstract generation model. Conclusion: The method proposed in this paper is effective, and the performance has been greatly improved on the basis of the benchmark model. Compared with the existing Chinese abstract generation model, its performance is also superior. In the next step, more visual prior information will be incorporated into the model, such as the action category, the relationship between the object and the object, etc., to further improve the quality of the description sentence, and achieve the effect of “seeing the picture writing”.

Multi-attention mechanism for Chinese description of videos

An Unsupervised Video Summarization Method Based on Multimodal Representation.

A Video Description Model with Improved Attention Mechanism

Deep Hierarchical Attention Network for Video Description

Attention-Based Multimodal Fusion for Video Description

Application of Dual Attention Mechanism in Chinese Image Captioning

Multimodal-enhanced hierarchical attention network for video captioning

A Dataset with Multi-Modal Information and Multi-Granularity Descriptions for Video Captioning

Bidirectional Long-Short Term Memory for Video Description

Multi-modal Summarization for Video-containing Documents

Multiple cross-attention for video-subtitle moment retrieval

Coarse-to-fine dual-level attention for video-text cross modal retrieval

Multi-Modal interpretable automatic video captioning

Research on Volleyball Video Intelligent Description Technology Combining the Long-Term and Short-Term Memory Network and Attention Mechanism

Multimodal Semantic Attention Network for Video Captioning

GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization

Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention

Multi-scale features with temporal information guidance for video captioning

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

Transformer-Based Interactive Multi-Modal Attention Network for Video Sentiment Detection

Towards Holistic Language-video Representation: the language model-enhanced MSR-Video to Text Dataset