Abstract:Video captioning focuses on generating natural language descriptions according to the video content. Existing works mainly explore this multimodal learning with the paired source video and corresponding sentence, which have achieved competitive performances. Nonetheless, learning from video-description pair cannot capture implicit external knowledge, i.e. multiple visual context information and linguistic clues existing in the video-language dataset, which may limit the cognitive capability of the model to generate diverse descriptions. To this end, we propose a Memory-based Augmentation Network (MAN), in which a memory structure is designed to augment the current encoder-decoder framework by incorporating implicit external knowledge with a neural memory. Specifically, we first propose a visual memory for the encoder to store multiple visual contexts across videos in the dataset, which is utilized to obtain memory-augmented contextual features for the source video. In addition, a textual memory is introduced for the decoder to capture the external language clues across sentences in the dataset. It is adapted to capture memory-augmented language features in each time step. The proposed approach is able to capture comprehensive contextual understanding compared to the basic encoder-decoder framework, which is more compatible with the human cognitive process. Extensive experiments on three video captioning datasets including MSVD, MSR-VTT, and VATEX demonstrate the effectiveness of the proposed method. The source code is available at https://github.com/zchoi/MAN .

Sentence Simplification with Memory-Augmented Neural Networks.

A Constrained Sequence-to-Sequence Neural Model for Sentence Simplification.

Improving sentence simplification model with ordered neurons network

Neural Sentence Simplification with Semantic Dependency Information

Automatic Document Summarization Via Deep Neural Networks

Dynamic Multi-Level Multi-Task Learning for Sentence Simplification

An Experimental Study of LSTM Encoder-Decoder Model for Text Simplification

Sentence Simplification with Deep Reinforcement Learning

BoschAI @ PLABA 2023: Leveraging Edit Operations in End-to-End Neural Sentence Simplification

Towards Arabic Sentence Simplification via Classification and Generative Approaches

Unsupervised Sentence Simplification via Dependency Parsing

Natural Language Inference Using Lstm Model With Sentence Fusion

Memory-Based Augmentation Network for Video Captioning

Semi-Supervised Text Simplification with Back-Translation and Asymmetric Denoising Autoencoders

Complexity-Weighted Loss and Diverse Reranking for Sentence Simplification

Memory-augmented Neural Machine Translation.

Similarity-aware Neural Machine Translation: Reducing Human Translator Efforts by Leveraging High-Potential Sentences with Translation Memory

A Semantic Relevance Based Neural Network for Text Summarization and Text Simplification

Structural-semantics Guided Program Simplification for Understanding Neural Code Intelligence Models.

A Syntax-Augmented and Headline-Aware Neural Text Summarization Method

A Novel Sentence-Level Agreement Architecture for Neural Machine Translation