Abstract:Image description task is the intersection of computer vision and natural language processing, and it has important prospects, including helping computers understand images and obtaining information for the visually impaired. This study presents an innovative approach employing deep reinforcement learning to enhance the accuracy of natural language descriptions of images. Our method focuses on refining the reward function in deep reinforcement learning, facilitating the generation of precise descriptions by aligning visual and textual features more closely. Our approach comprises three key architectures. Firstly, it utilizes Residual Network 101 (ResNet-101) and Faster Region-based Convolutional Neural Network (Faster R-CNN) to extract average and local image features, respectively, followed by the implementation of a dual attention mechanism for intricate feature fusion. Secondly, the Transformer model is engaged to derive contextual semantic features from textual data. Finally, the generation of descriptive text is executed through a two-layer long short-term memory network (LSTM), directed by the value and reward functions. Compared with the image description method that relies on deep learning, the score of Bilingual Evaluation Understudy (BLEU-1) is 0.762, which is 1.6% higher, and the score of BLEU-4 is 0.299. Consensus-based Image Description Evaluation (CIDEr) scored 0.998, Recall-Oriented Understudy for Gisting Evaluation (ROUGE) scored 0.552, the latter improved by 0.36%. These results not only attest to the viability of our approach but also highlight its superiority in the realm of image description. Future research can explore the integration of our method with other artificial intelligence (AI) domains, such as emotional AI, to create more nuanced and context-aware systems.

Improvement of image description using bidirectional LSTM

Image Captioning with Deep Bidirectional LSTMs

Deep Sequential Fusion LSTM Network for Image Description

LSTM-in-LSTM for Generating Long Descriptions of Images.

Image Captioning with Bidirectional Semantic Attention-Based Guiding of Long Short-Term Memory

Describing Video with Attention-Based Bidirectional LSTM

Bidirectional Long-Short Term Memory for Video Description

A novel Multi-Layer Attention Framework for visual description prediction using bidirectional LSTM

Enhancing Image Description Generation through Deep Reinforcement Learning: Fusing Multiple Visual Features and Reward Mechanisms

Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering

Image Captioning with Deep LSTM Based on Sequential Residual.

Visual Attention Based on Long-Short Term Memory Model for Image Caption Generation

Image Caption Generation Using Contextual Information Fusion with Bi-LSTM-s

Video Description using Bidirectional Recurrent Neural Networks

Generate Image Descriptions based on Deep RNN and Memory Cells for Images Features

CaptionNet: A Tailor-made Recurrent Neural Network for Generating Image Descriptions

Multi-modal gated recurrent units for image description

Fast Image Captioning Using LSTM

BENet: bi-directional enhanced network for image captioning

Rich Visual and Language Representation with Complementary Semantics for Video Captioning

CC-LSTM: Cross and Conditional Long-Short Time Memory for Video Captioning