Abstract:Recent progress in stylized image captioning has been achieved through the encoder-decoder framework that generates a sentence in one-pass decoding process. However, it remains difficult for such a decoding process to simultaneously capture the syntactic structure, infer the semantic concepts and express the linguistic styles. Research in psycholinguistics has revealed that the language production process of humans involves multiple stages, starting with several rough concepts and ending with fluent sentences. With this in mind, we propose a novel stylized image captioning approach that generates stylized sentences in a multi-pass decoding process by training three cooperative neural modules under the reinforcement learning paradigm. A low-level neural module called syntax module first generates the overall syntactic structure of the stylized sentence. Next, two high-level neural modules, namely concept module and style module , incorporate the words that describe factual content and the words that express linguistic style, respectively. Since the three modules contribute to different aspects of the stylized sentence, i.e. the fluency, the relevancy of the factual content and the style accuracy, we encourage the modules to specialize in their own tasks by designing different rewards for different actions. We also design an attention mechanism to facilitate the communication between the high-level and low-level modules. With the help of the attention mechanism, the high-level modules are able to take the global structure of the sentence into consideration and maintain the consistency between the factual content and the linguistic style. Evaluations on several public benchmark datasets demonstrate that our method outperforms the existing one-pass decoding methods in terms of multiple different evaluation metrics.

Curriculum pre-training for stylized neural machine translation

Improving Stylized Neural Machine Translation with Iterative Dual Knowledge Transfer.

UATST: Towards Unpaired Arbitrary Text-Guided Style Transfer with Cross-Space Modulation

Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models

Token-wise Curriculum Learning for Neural Machine Translation

Norm-Based Curriculum Learning for Neural Machine Translation

Data Selection Curriculum for Neural Machine Translation

A Multi-task Multi-stage Transitional Training Framework for Neural Chat Translation

Curriculum Pre-training for End-to-End Speech Translation

Self-Guided Curriculum Learning for Neural Machine Translation

Synthetic Pre-Training Tasks for Neural Machine Translation

Deep Fusing Pre-trained Models into Neural Machine Translation.

Pre-training for Speech Translation: CTC Meets Optimal Transport

Reinforcement Learning based Curriculum Optimization for Neural Machine Translation

Controlling Styles in Neural Machine Translation with Activation Prompt

Low-Resource Neural Machine Translation Using XLNet Pre-training Model

Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation

Towards Robust Neural Machine Translation with Iterative Scheduled Data-Switch Training.

Scheduled Multi-task Learning for Neural Chat Translation

Learning Cooperative Neural Modules for Stylized Image Captioning

Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation