Abstract:Image description task is the intersection of computer vision and natural language processing, and it has important prospects, including helping computers understand images and obtaining information for the visually impaired. This study presents an innovative approach employing deep reinforcement learning to enhance the accuracy of natural language descriptions of images. Our method focuses on refining the reward function in deep reinforcement learning, facilitating the generation of precise descriptions by aligning visual and textual features more closely. Our approach comprises three key architectures. Firstly, it utilizes Residual Network 101 (ResNet-101) and Faster Region-based Convolutional Neural Network (Faster R-CNN) to extract average and local image features, respectively, followed by the implementation of a dual attention mechanism for intricate feature fusion. Secondly, the Transformer model is engaged to derive contextual semantic features from textual data. Finally, the generation of descriptive text is executed through a two-layer long short-term memory network (LSTM), directed by the value and reward functions. Compared with the image description method that relies on deep learning, the score of Bilingual Evaluation Understudy (BLEU-1) is 0.762, which is 1.6% higher, and the score of BLEU-4 is 0.299. Consensus-based Image Description Evaluation (CIDEr) scored 0.998, Recall-Oriented Understudy for Gisting Evaluation (ROUGE) scored 0.552, the latter improved by 0.36%. These results not only attest to the viability of our approach but also highlight its superiority in the realm of image description. Future research can explore the integration of our method with other artificial intelligence (AI) domains, such as emotional AI, to create more nuanced and context-aware systems.

Intelligent image automatic description method based on deep neural network

The image annotation algorithm using convolutional features from intermediate layer of deep learning

Image Caption Description of Traffic Scene Based on Deep Learning

The Image Annotation Method by Convolutional Features from Intermediate Layer of Deep Learning Based on Internet of Things

Deep Sequential Fusion LSTM Network for Image Description

A Novel Automatic Image Annotation Method Based on Co-occurrence Keywords

Research on the Visual Image Processing Based on Neural Network

Semantic Communication Approach for Multi-Task Image Transmission

Hierarchical Deep Neural Network for Image Captioning

Automatic Image Annotation Combining Semantic Neighbors and Deep Features

Automatically Generating Natural Language Descriptions of Images by a Deep Hierarchical Framework.

Automatic annotation of visual deep neural network

Enhancing Image Description Generation through Deep Reinforcement Learning: Fusing Multiple Visual Features and Reward Mechanisms

A deep neural network based feature fusion panoramic segmentation method

Generation method of image description from structured text

Automatic annotation of visual deep neural networks

Image Emotion Recognition Based on Deep Neural Network

Automatic Tagging by Leveraging Visual and Annotated Features in Social Media

Image Semantic Description and Automatic Semantic Annotation

NATURAL LANGUAGE DESCRIPTION OF REMOTE SENSING IMAGES BASED ON DEEP LEARNING

A Multi-feature Fusion and SSAE-Based Deep Network for Image Semantic Recognition