Abstract:Image caption technology aims to convert visual features of images, extracted by computers, into meaningful semantic information. Therefore, the computers can generate text descriptions that resemble human perception, enabling tasks such as image classification, retrieval, and analysis. In recent years, the performance of image caption has been significantly enhanced with the introduction of encoder-decoder architecture in machine translation and the utilization of deep neural networks. However, several challenges still persist in this domain. Therefore, this paper proposes a novel method to address the issue of visual information loss and non-dynamic adjustment of input images during decoding. We introduce a guided decoding network that establishes a connection between the encoding and decoding parts. Through this connection, encoding information can provide guidance to the decoding process, facilitating automatic adjustment of the decoding information. In addition, Dense Convolutional Network (DenseNet) and Multiple Instance Learning (MIL) are adopted in the image encoder, and Nested Long Short-Term Memory (NLSTM) is utilized as the decoder to enhance the extraction and parsing capability of image information during the encoding and decoding process. In order to further improve the performance of our image caption model, this study incorporates an attention mechanism to focus details and constructs a double-layer decoding structure, which facilitates the enhancement of the model in terms of providing more detailed descriptions and enriched semantic information. Furthermore, the Deep Reinforcement Learning (DRL) method is employed to train the model by directly optimizing the identical set of evaluation indexes, which solves the problem of inconsistent training and evaluation standards. Finally, the model is trained and tested on MS COCO and Flickr 30 k datasets, and the results show that the model has improved compared with commonly used models in the evaluation indicators such as BLEU, METEOR and CIDEr.

Infrared Image Captioning Based on Unsupervised Learning and Reinforcement Learning

Structured Label Inference for Visual Understanding.

Infrared-LLaVA: Enhancing Understanding of Infrared Images in Multi-Modal Large Language Models

Automatic Captioning Based on Visible and Infrared Images

Infrared Image Captioning with Wearable Device.

Image Captioning with Partially Rewarded Imitation Learning.

Remote sensing image captioning via Variational Autoencoder and Reinforcement Learning

Enhancing Image Description Generation through Deep Reinforcement Learning: Fusing Multiple Visual Features and Reward Mechanisms

Image Captioning based on Deep Reinforcement Learning

A Semi-supervised Framework for Image Captioning

Explicit Image Caption Reasoning: Generating Accurate and Informative Captions for Complex Scenes with LMM

An image caption model based on attention mechanism and deep reinforcement learning

DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding

Infrared Image Generation Based on Visual State Space and Contrastive Learning

See or Guess: Counterfactually Regularized Image Captioning

Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data

Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering

Region-Aware Image Captioning Via Interaction Learning

Tell and Guess: Cooperative Learning for Natural Image Caption Generation with Hierarchical Refined Attention

CapHDR2IR: Caption-Driven Transfer from Visible Light to Infrared Domain

Semi-supervised learning for image annotation based on conditional random fields