Abstract:Image caption technology aims to convert visual features of images, extracted by computers, into meaningful semantic information. Therefore, the computers can generate text descriptions that resemble human perception, enabling tasks such as image classification, retrieval, and analysis. In recent years, the performance of image caption has been significantly enhanced with the introduction of encoder-decoder architecture in machine translation and the utilization of deep neural networks. However, several challenges still persist in this domain. Therefore, this paper proposes a novel method to address the issue of visual information loss and non-dynamic adjustment of input images during decoding. We introduce a guided decoding network that establishes a connection between the encoding and decoding parts. Through this connection, encoding information can provide guidance to the decoding process, facilitating automatic adjustment of the decoding information. In addition, Dense Convolutional Network (DenseNet) and Multiple Instance Learning (MIL) are adopted in the image encoder, and Nested Long Short-Term Memory (NLSTM) is utilized as the decoder to enhance the extraction and parsing capability of image information during the encoding and decoding process. In order to further improve the performance of our image caption model, this study incorporates an attention mechanism to focus details and constructs a double-layer decoding structure, which facilitates the enhancement of the model in terms of providing more detailed descriptions and enriched semantic information. Furthermore, the Deep Reinforcement Learning (DRL) method is employed to train the model by directly optimizing the identical set of evaluation indexes, which solves the problem of inconsistent training and evaluation standards. Finally, the model is trained and tested on MS COCO and Flickr 30 k datasets, and the results show that the model has improved compared with commonly used models in the evaluation indicators such as BLEU, METEOR and CIDEr.

Image-to-Markup Generation with Coarse-to-Fine Attention

Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge.

An Attention Based Image To Latex Markup Decoder

Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition.

Text-to-Image Generation with Attention Based Recurrent Neural Networks

Image-to-LaTeX Converter for Mathematical Formulas and Text

Developing a seq2seq neural network using visual attention to transform mathematical expressions from images to LaTeX.

A GRU-based Encoder-Decoder Approach with Attention for Online Handwritten Mathematical Expression Recognition

Image to LaTeX with Graph Neural Network for Mathematical Formula Recognition

Translating math formula images to LaTeX sequences using deep neural networks with sequence-level training

Markup-to-Image Diffusion Models with Scheduled Sampling

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Contrast-augmented Diffusion Model with Fine-grained Sequence Alignment for Markup-to-Image Generation

Image caption generation with dual attention mechanism

Watch, Attend and Parse: an End-to-end Neural Network Based Approach to Handwritten Mathematical Expression Recognition.

Handwritten Mathematical Expression Recognition via Attention Aggregation Based Bi-directional Mutual Learning

An image caption model based on attention mechanism and deep reinforcement learning

ConvMath : A Convolutional Sequence Network for Mathematical Expression Recognition

Adaptively Aligned Image Captioning via Adaptive Attention Time

MarkovGen: Structured Prediction for Efficient Text-to-Image Generation