Abstract:In recent years, the field of Natural Language Generation (NLG) has been boosted by the recent advances in deep learning technologies. Nonetheless, these new data-intensive methods introduce language-dependent disparities in NLG as the main training data sets are in English. Also, most neural NLG systems use decoder-only (causal) transformer language models, which work well for English, but were not designed with other languages in mind. In this work we depart from the hypothesis that they may introduce generation bias in target languages with less rigid word ordering, subject omission, or different attachment preferences for relative clauses, so that for these target languages other language generation strategies may be more desirable. This paper first compares causal and non-causal language modeling for English and Spanish, two languages with different grammatical structures and over 1.5 billion and 0.5 billion speakers, respectively. For this purpose, we define a novel metric of average causal and non-causal context-conditioned entropy of the grammatical category distribution for both languages as an information-theoretic a priori approach. The evaluation of natural text sources (such as training data) in both languages reveals lower average non-causal conditional entropy in Spanish and lower causal conditional entropy in English. According to this experiment, Spanish is more predictable than English given a non-causal context. Then, by applying a conditional relative entropy metric to text generation experiments, we obtain as insights that the best performance is respectively achieved with causal NLG in English, and with non-causal NLG in Spanish. These insights support further research in NLG in Spanish using bidirectional transformer language models.

On the Entropy of Written Spanish

A New Look at the Classical Entropy of Written English

Complexity measurement of natural and artificial languages

A software for learning Information Theory basics with emphasis on Entropy of Spanish

Entropy in Different Text Types.

Quantitative Entropy Study of Language Complexity

Word-length Entropies and Correlations of Natural Language Written Texts

Entropy and type-token ratio in gigaword corpora

Quantifying literature quality using complexity criteria

ENTROPIC ANALYSIS OF THE ROLE OF WORDS IN LITERARY TEXTS

Prediction and Entropy of Printed English

Characterizing the social diversity of bilingualism using language entropy

Entropy of a Zipfian Distributed Lexicon

An Estimate Method Of The Minimum Entropy Of Natural Languages

Entropy and Long range correlations in literary English

Estimating the Minimum Entropy of Chinese and Japanese Languages.

Predictability and Causality in Spanish and English Natural Language Generation

Fast calculation of entropy with Zhang's estimator

Character Entropy in Modern and Historical Texts: Comparison Metrics for an Undeciphered Manuscript

Rank diversity of languages: Generic behavior in computational linguistics

Language Statistics at Different Spatial, Temporal, and Grammatical Scales