Abstract:Objective: The Chinese description of images combines the two directions of computer vision and natural language processing. It is a typical representative of multi-mode and cross-domain problems with artificial intelligence algorithms. The image Chinese description model needs to output a Chinese description for each given test picture, describe the sentence requirements to conform to the natural language habits, and point out the important information in the image, covering the main characters, scenes, actions and other content. Since the current open source datasets are mostly in English, the research on the direction of image description is mainly in English. Chinese descriptions usually have greater flexibility in syntax and lexicalization, and the challenges of algorithm implementation are also large. Therefore, only a few people have studied image descriptions, especially Chinese descriptions. Methods: This study attempts to derive a model of image description generation from the Flickr8k-cn and Flickr30k-cn datasets. At each time period of the description, the model can decide whether to rely more on images or text information. The model captures more important information from the image to improve the richness and accuracy of the Chinese description of the image. The image description data set of this study is mainly composed of Chinese description sentences. The method consists of an encoder and a decoder. The encoder is based on a convolutional neura l network. The decoder is based on a long-short memory network and is composed of a multi-modal summary generation network. Results: Experiments on Flickr8k-cn and Flickr30k-cn Chinese datasets show that the proposed method is superior to the existing Chinese abstract generation model. Conclusion: The method proposed in this paper is effective, and the performance has been greatly improved on the basis of the benchmark model. Compared with the existing Chinese abstract generation model, its performance is also superior. In the next step, more visual prior information will be incorporated into the model, such as the action category, the relationship between the object and the object, etc., to further improve the quality of the description sentence, and achieve the effect of “seeing the picture writing”.

Recurrent Attention LSTM Model for Image Chinese Caption Generation

Recurrent convolutional video captioning with global and local attention.

Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering

Chinese Image Caption Generation via Visual Attention and Topic Modeling

An image caption model based on attention mechanism and deep reinforcement learning

Text-to-Image Generation with Attention Based Recurrent Neural Networks

Intelligent image captioning

Chinese image caption of Inceptionv4 and double-layer GRUs based on attention mechanism

Recurrent Attention and Semantic Gate for Remote Sensing Image Captioning

Image caption generation with dual attention mechanism

Time-Dependent Pre-Attention Model For Image Captioning

RNIC-A retrospect network for image captioning

Reference Based LSTM for Image Captioning.

A Parallel-Fusion RNN-LSTM Architecture for Image Caption Generation

Improve Image Captioning By Self-Attention

AttResNet: Attention-based ResNet for Image Captioning

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Image Captioning with Deep Bidirectional LSTMs

Fast Image Captioning Using LSTM

Visual enhanced gLSTM for image captioning

Application of Dual Attention Mechanism in Chinese Image Captioning