Abstract:Neural network language models (LMs) are shown to be effective in improving the performance of statistical machine translation (SMT) systems. However, state-of-the-art neural network LMs usually use words before the current position as context and neglect global topic information, which can helpmachine translation (MT) systems to select better translation candidates from a higher perspective. In this work, we propose improvement of the state-of-the-art feedforward neural language model with topic information. Two main issues need to be tackled when adding topics into neural network LMs for SMT: one is how to incorporate topics to the neural network; the other is how to get target-side topic distribution before translation. We incorporate topics by appending topic distribution to the input layer of a feedforward LM. We adopt a multinomial logistic-regression (MLR) model to predict the target-side topic distribution based on source side information. Moreover, we propose a feedforward neural network model to learn joint representations on the source side for topic prediction. LM experiments demonstrate that the perplexity on validation set can be greatly reduced by the topic-enhanced feedforward LM, and the prediction of target-side topics can be improved dramatically with the MLR model equipped with the joint source representations. A final MT experiment, conducted on a large-scale Chinese-English dataset, shows that our feedforward LM with predicted topics improves the translation performance against a strong baseline.

Revisiting Topic-Guided Language Models

Topic Discovery Based on LDA_col Model and Topic Significance Re-ranking.

Neural Topic Model with Reinforcement Learning.

Revisiting Automated Topic Model Evaluation with Large Language Models

Adaptation of Language Models for SMT Using Neural Networks with Topic Information.

Topic Modeling Revisited: A Document Graph-based Neural Network Perspective

Revisiting K-Nn for Fine-Tuning Pre-trained Language Models

Revisiting Simple Neural Probabilistic Language Models

Prompting Large Language Models for Topic Modeling

Variational Gaussian Topic Model with Invertible Neural Projections

Extracting nonlinear neural topics with neural variational bayes

LTSG: Latent Topical Skip-Gram for Mutually Improving Topic Model and Vector Representations

Neural Topic Modeling with Large Language Models in the Loop

TopicGPT: A Prompt-based Topic Modeling Framework

Language Model Evaluation Beyond Perplexity

Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling

A Topic-Triggered Language Model for Statistical Machine Translation.

Learning Multilingual Topics with Neural Variational Inference

Paragraph Vector Based Topic Model for Language Model Adaptation.

Neural Topic Modeling Based on Cycle Adversarial Training and Contrastive Learning.

Explainable and Discourse Topic-aware Neural Language Understanding