Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche

Christophe Coupé,Yoon Mi Oh,Dan Dediu,François Pellegrino

DOI: https://doi.org/10.1126/sciadv.aaw2594

IF: 13.6

2019-09-06

Science Advances

Abstract:Human languages encode similar average information rates (~39 bits/s) despite their remarkable differences.

multidisciplinary sciences

What problem does this paper attempt to address?

Human languages with greater information density have higher communication speed but lower conversation breadth

Pedro Aceves,James A. Evans

DOI: https://doi.org/10.1038/s41562-024-01815-w

IF: 24.252

2024-02-17

Nature Human Behaviour

Abstract:Human languages vary widely in how they encode information within circumscribed semantic domains (for example, time, space, colour, human body parts and activities), but little is known about the global structure of semantic information and nothing about its relation to human communication. We first show that across a sample of ~1,000 languages, there is broad variation in how densely languages encode information into words. Second, we show that this language information density is associated with a denser configuration of semantic information. Finally, we trace the relationship between language information density and patterns of communication, showing that informationally denser languages tend towards faster communication but conceptually narrower conversations or expositions within which topics are discussed at greater depth. These results highlight an important source of variation across the human communicative channel, revealing that the structure of language shapes the nature and texture of human engagement, with consequences for human behaviour across levels of society.

psychology, experimental,neurosciences,multidisciplinary sciences
Human languages order information efficiently

Daniel Gildea,T. Florian Jaeger

DOI: https://doi.org/10.48550/arXiv.1510.02823

2015-10-10

Abstract:Most languages use the relative order between words to encode meaning relations. Languages differ, however, in what orders they use and how these orders are mapped onto different meanings. We test the hypothesis that, despite these differences, human languages might constitute different `solutions' to common pressures of language use. Using Monte Carlo simulations over data from five languages, we find that their word orders are efficient for processing in terms of both dependency length and local lexical probability. This suggests that biases originating in how the brain understands language strongly constrain how human languages change over generations.

Computation and Language
Compression and the origins of Zipf's law of abbreviation

R. Ferrer-i-Cancho,C. Bentz,C. Seguin

DOI: https://doi.org/10.48550/arXiv.1504.04884

2016-05-04

Abstract:Languages across the world exhibit Zipf's law of abbreviation, namely more frequent words tend to be shorter. The generalized version of the law - an inverse relationship between the frequency of a unit and its magnitude - holds also for the behaviours of other species and the genetic code. The apparent universality of this pattern in human language and its ubiquity in other domains calls for a theoretical understanding of its origins. To this end, we generalize the information theoretic concept of mean code length as a mean energetic cost function over the probability and the magnitude of the types of the repertoire. We show that the minimization of that cost function and a negative correlation between probability and the magnitude of types are intimately related.

Information Theory,Computation and Language,Social and Information Networks,Data Analysis, Statistics and Probability
How Efficiency Shapes Human Language

Edward Gibson,Richard Futrell,Steven P Piantadosi,Isabelle Dautriche,Kyle Mahowald,Leon Bergen,Roger Levy,Steven P. Piantadosi

DOI: https://doi.org/10.1016/j.tics.2019.02.003

IF: 19.9

2019-05-01

Trends in Cognitive Sciences

Abstract:<p>Cognitive science applies diverse tools and perspectives to study human language. Recently, an exciting body of work has examined linguistic phenomena through the lens of efficiency in usage: what otherwise puzzling features of language find explanation in formal accounts of how language might be optimized for communication and learning? Here, we review studies that deploy formal tools from probability and information theory to understand how and why language works the way that it does, focusing on phenomena ranging from the lexicon through syntax. These studies show how a pervasive pressure for efficiency guides the forms of natural language and indicate that a rich future for language research lies in connecting linguistics to cognitive psychology and mathematical theories of communication and inference.</p>

behavioral sciences,psychology, experimental,neurosciences
Not lost in translation: neural responses shared across languages

Christopher J Honey,Christopher R Thompson,Yulia Lerner,Uri Hasson

DOI: https://doi.org/10.1523/JNEUROSCI.1800-12.2012

2012-10-31

Abstract:How similar are the brains of listeners who hear the same content expressed in different languages? We directly compared the fMRI response time courses of English speakers and Russian speakers who listened to a real-life Russian narrative and its English translation. In the translation, we tried to preserve the content of the narrative while reducing the structural similarities across languages. The story evoked similar brain responses, invariant to the structural changes across languages, beginning just outside early auditory areas and extending through temporal, parietal, and frontal cerebral cortices. The similarity of responses across languages was nearly equal to the similarity of responses within each language group. The present results demonstrate that the human brain processes real-life information in a manner that is largely insensitive to the language in which that information is conveyed. The methods introduced here can potentially be used to quantify the transmission of meaning across cultural and linguistic boundaries.
Evolution of Efficient Symbolic Communication Codes

Anton Kolonin

2023-06-11

Abstract:The paper explores how the human natural language structure can be seen as a product of evolution of inter-personal communication code, targeting maximisation of such culture-agnostic and cross-lingual metrics such as anti-entropy, compression factor and cross-split F1 score. The exploration is done as part of a larger unsupervised language learning effort, the attempt is made to perform meta-learning in a space of hyper-parameters maximising F1 score based on the "ground truth" language structure, by means of maximising the metrics mentioned above. The paper presents preliminary results of cross-lingual word-level segmentation tokenisation study for Russian, Chinese and English as well as subword segmentation or morphological parsing study for English. It is found that language structure form the word-level segmentation or tokenisation can be found as driven by all of these metrics, anti-entropy being more relevant to English and Russian while compression factor more specific for Chinese. The study for subword segmentation or morphological parsing on English lexicon has revealed straight connection between the compression been found to be associated with compression factor, while, surprising, the same connection with anti-entropy has turned to be the inverse.

Computation and Language,Information Theory,Machine Learning
Discrete Messages Improve Communication Efficiency among Isolated Intelligent Agents

Hang Chen,Yuchuan Jang,Weijie Zhou,Cristian Meo,Ziwei Chen,Dianbo Liu

2024-10-18

Abstract:Individuals, despite having varied life experiences and learning processes, can communicate effectively through languages. This study aims to explore the efficiency of language as a communication medium. We put forth two specific hypotheses: First, discrete messages are more effective than continuous ones when agents have diverse personal experiences. Second, communications using multiple discrete tokens are more advantageous than those using a single token. To valdate these hypotheses, we designed multi-agent machine learning experiments to assess communication efficiency using various information transmission methods between speakers and listeners. Our empirical findings indicate that, in scenarios where agents are exposed to different data, communicating through sentences composed of discrete tokens offers the best inter-agent communication efficiency. The limitations of our finding include lack of systematic advantages over other more sophisticated encoder-decoder model such as variational autoencoder and lack of evluation on non-image dataset, which we will leave for future studies.

Machine Learning,Information Theory
Evaluating performance of neural codes in model neural communication networks

Chris G Antonopoulos,Ezequiel Bianco-Martinez,Murilo S Baptista,Chris G. Antonopoulos,Murilo S. Baptista

DOI: https://doi.org/10.1016/j.neunet.2018.10.008

IF: 7.8

2019-01-01

Neural Networks

Abstract:Information needs to be appropriately encoded to be reliably transmitted over physical media. Similarly, neurons have their own codes to convey information in the brain. Even though it is well-known that neurons exchange information using a pool of several protocols of spatio-temporal encodings, the suitability of each code and their performance as a function of network parameters and external stimuli is still one of the great mysteries in neuroscience. This paper sheds light on this by modeling small-size networks of chemically and electrically coupled Hindmarsh-Rose spiking neurons. We focus on a class of temporal and firing-rate codes that result from neurons' membrane-potentials and phases, and quantify numerically their performance estimating the Mutual Information Rate, aka the rate of information exchange. Our results suggest that the firing-rate and interspike-intervals codes are more robust to additive Gaussian white noise. In a network of four interconnected neurons and in the absence of such noise, pairs of neurons that have the largest rate of information exchange using the interspike-intervals and firing-rate codes are not adjacent in the network, whereas spike-timings and phase codes (temporal) promote large rate of information exchange for adjacent neurons. If that result would have been possible to extend to larger neural networks, it would suggest that small microcircuits would preferably exchange information using temporal codes (spike-timings and phase codes), whereas on the macroscopic scale, where there would be typically pairs of neurons not directly connected due to the brain's sparsity, firing-rate and interspike-intervals codes would be the most efficient codes.

computer science, artificial intelligence,neurosciences
A surprisal--duration trade-off across and within the world's languages

Tiago Pimentel,Clara Meister,Elizabeth Salesky,Simone Teufel,Damián Blasi,Ryan Cotterell

DOI: https://doi.org/10.48550/arXiv.2109.15000

2021-09-30

Computation and Language

Abstract:While there exist scores of natural languages, each with its unique features and idiosyncrasies, they all share a unifying theme: enabling human communication. We may thus reasonably predict that human cognition shapes how these languages evolve and are used. Assuming that the capacity to process information is roughly constant across human populations, we expect a surprisal--duration trade-off to arise both across and within languages. We analyse this trade-off using a corpus of 600 languages and, after controlling for several potential confounds, we find strong supporting evidence in both settings. Specifically, we find that, on average, phones are produced faster in languages where they are less surprising, and vice versa. Further, we confirm that more surprising phones are longer, on average, in 319 languages out of the 600. We thus conclude that there is strong evidence of a surprisal--duration trade-off in operation, both across and within the world's languages.
Zipf’s Law: Balancing Signal Usage Cost and Communication Efficiency

Christoph Salge,Nihat Ay,Daniel Polani,Mikhail Prokopenko

DOI: https://doi.org/10.1371/journal.pone.0139475

IF: 3.7

2015-10-01

PLoS ONE

Abstract:We propose a model that explains the reliable emergence of power laws (e.g., Zipf's law) during the development of different human languages. The model incorporates the principle of least effort in communications, minimizing a combination of the information-theoretic communication inefficiency and direct signal cost. We prove a general relationship, for all optimal languages, between the signal cost distribution and the resulting distribution of signals. Zipf's law then emerges for logarithmic signal cost distributions, which is the cost distribution expected for words constructed from letters or phonemes.

multidisciplinary sciences
Information content versus word length in natural language: A reply to Ferrer-i-Cancho and Moscoso del Prado Martin [<a class="link-https" data-arxiv-id="1209.1751" href="https://arxiv.org/abs/1209.1751">arXiv:1209.1751</a>]

Steven T. Piantadosi,Harry Tily,Edward Gibson

DOI: https://doi.org/10.48550/arXiv.1307.6726

2013-07-25

Abstract:Recently, Ferrer i Cancho and Moscoso del Prado Martin [<a class="link-https" data-arxiv-id="1209.1751" href="https://arxiv.org/abs/1209.1751">arXiv:1209.1751</a>] argued that an observed linear relationship between word length and average surprisal (Piantadosi, Tily, & Gibson, 2011) is not evidence for communicative efficiency in human language. We discuss several shortcomings of their approach and critique: their model critically rests on inaccurate assumptions, is incapable of explaining key surprisal patterns in language, and is incompatible with recent behavioral results. More generally, we argue that statistical models must not critically rely on assumptions that are incompatible with the real system under study.

Computation and Language,Probability,Data Analysis, Statistics and Probability
Compression as a universal principle of animal behavior

R. Ferrer-i-Cancho,A. Hernández-Fernández,D. Lusseau,G. Agoramoorthy,M. J. Hsu,S. Semple

DOI: https://doi.org/10.1111/cogs.12061

2013-03-25

Abstract:A key aim in biology and psychology is to identify fundamental principles underpinning the behavior of animals, including humans. Analyses of human language and the behavior of a range of non-human animal species have provided evidence for a common pattern underlying diverse behavioral phenomena: words follow Zipf's law of brevity (the tendency of more frequently used words to be shorter), and conformity to this general pattern has been seen in the behavior of a number of other animals. It has been argued that the presence of this law is a sign of efficient coding in the information theoretic sense. However, no strong direct connection has been demonstrated between the law and compression, the information theoretic principle of minimizing the expected length of a code. Here we show that minimizing the expected code length implies that the length of a word cannot increase as its frequency increases. Furthermore, we show that the mean code length or duration is significantly small in human language, and also in the behavior of other species in all cases where agreement with the law of brevity has been found. We argue that compression is a general principle of animal behavior, that reflects selection for efficiency of coding.

Neurons and Cognition,Computation and Language,Information Theory,Data Analysis, Statistics and Probability,Quantitative Methods
How Far can Neural Correlations Reduce Uncertainty? Comparison of Information Transmission Rates for Markov and Bernoulli Processes

Agnieszka Pregowska,Ehud Kaplan,Janusz Szczepanski

DOI: https://doi.org/10.1142/S0129065719500035

Abstract:The nature of neural codes is central to neuroscience. Do neurons encode information through relatively slow changes in the firing rates of individual spikes (rate code) or by the precise timing of every spike (temporal code)? Here we compare the loss of information due to correlations for these two possible neural codes. The essence of Shannon's definition of information is to combine information with uncertainty: the higher the uncertainty of a given event, the more information is conveyed by that event. Correlations can reduce uncertainty or the amount of information, but by how much? In this paper we address this question by a direct comparison of the information per symbol conveyed by the words coming from a binary Markov source (temporal code) with the information per symbol coming from the corresponding Bernoulli source (uncorrelated, rate code). In a previous paper we found that a crucial role in the relation between information transmission rates (ITRs) and firing rates is played by a parameter s, which is the sum of transition probabilities from the no-spike state to the spike state and vice versa. We found that in this case too a crucial role is played by the same parameter s. We calculated the maximal and minimal bounds of the quotient of ITRs for these sources. Next, making use of the entropy grouping axiom, we determined the loss of information in a Markov source compared with the information in the corresponding Bernoulli source for a given word length. Our results show that in the case of correlated signals the loss of information is relatively small, and thus temporal codes, which are more energetically efficient, can replace rate codes effectively. These results were confirmed by experiments.
From "um" to "yeah": Producing, predicting, and regulating information flow in human conversation

Claire Augusta Bergey,Simon DeDeo

2024-03-14

Abstract:Conversation demands attention. Speakers must call words to mind, listeners must make sense of them, and both together must negotiate this flow of information, all in fractions of a second. We used large language models to study how this works in a large-scale dataset of English-language conversation, the CANDOR corpus. We provide a new estimate of the information density of unstructured conversation, of approximately 13 bits/second, and find significant effects associated with the cognitive load of both retrieving, and presenting, that information. We also reveal a role for backchannels -- the brief yeahs, uh-huhs, and mhmms that listeners provide -- in regulating the production of novelty: the lead-up to a backchannel is associated with declining information rate, while speech downstream rebounds to previous rates. Our results provide new insights into long-standing theories of how we respond to fluctuating demands on cognitive resources, and how we negotiate those demands in partnership with others.

Computation and Language,Information Theory,Neurons and Cognition
Information density as a predictor of communication dynamics

Gary Lupyan,Pablo Contreras Kallens,Rick Dale

DOI: https://doi.org/10.1016/j.tics.2024.03.012

IF: 19.9

2024-04-18

Trends in Cognitive Sciences

Abstract:In a recent paper, Aceves and Evans computed information and semantic density measures for hundreds of languages, and showed that these measures predict the pace and breadth of ideas in communication. Here, we summarize their key findings and situate them in a broader debate about the adaptive nature of language.

behavioral sciences,psychology, experimental,neurosciences
Revisiting Entropy Rate Constancy in Text

Vivek Verma,Nicholas Tomlin,Dan Klein

2023-10-18

Abstract:The uniform information density (UID) hypothesis states that humans tend to distribute information roughly evenly across an utterance or discourse. Early evidence in support of the UID hypothesis came from Genzel & Charniak (2002), which proposed an entropy rate constancy principle based on the probability of English text under n-gram language models. We re-evaluate the claims of Genzel & Charniak (2002) with neural language models, failing to find clear evidence in support of entropy rate constancy. We conduct a range of experiments across datasets, model sizes, and languages and discuss implications for the uniform information density hypothesis and linguistic theories of efficient communication more broadly.

Computation and Language
Towards Human-Agent Communication via the Information Bottleneck Principle

Mycal Tucker,Julie Shah,Roger Levy,Noga Zaslavsky

DOI: https://doi.org/10.48550/arXiv.2207.00088

IF: 14.4

2022-06-30

Artificial Intelligence

Abstract:Emergent communication research often focuses on optimizing task-specific utility as a driver for communication. However, human languages appear to evolve under pressure to efficiently compress meanings into communication signals by optimizing the Information Bottleneck tradeoff between informativeness and complexity. In this work, we study how trading off these three factors -- utility, informativeness, and complexity -- shapes emergent communication, including compared to human communication. To this end, we propose Vector-Quantized Variational Information Bottleneck (VQ-VIB), a method for training neural agents to compress inputs into discrete signals embedded in a continuous space. We train agents via VQ-VIB and compare their performance to previously proposed neural architectures in grounded environments and in a Lewis reference game. Across all neural architectures and settings, taking into account communicative informativeness benefits communication convergence rates, and penalizing communicative complexity leads to human-like lexicon sizes while maintaining high utility. Additionally, we find that VQ-VIB outperforms other discrete communication methods. This work demonstrates how fundamental principles that are believed to characterize human language evolution may inform emergent communication in artificial agents.
Information-Theoretic Limits of Strategic Communication

Mael Le Treust,Tristan Tomala

DOI: https://doi.org/10.48550/arXiv.1807.05147

2018-07-13

Abstract:In this article, we investigate strategic information transmission over a noisy channel. This problem has been widely investigated in Economics, when the communication channel is perfect. Unlike in Information Theory, both encoder and decoder have distinct objectives and choose their encoding and decoding strategies accordingly. This approach radically differs from the conventional Communication paradigm, which assumes transmitters are of two types: either they have a common goal, or they act as opponent, e.g. jammer, eavesdropper. We formulate a point-to-point source-channel coding problem with state information, in which the encoder and the decoder choose their respective encoding and decoding strategies in order to maximize their long-run utility functions. This strategic coding problem is at the interplay between Wyner-Ziv's scenario and the Bayesian persuasion game of Kamenica-Gentzkow. We characterize a single-letter solution and we relate it to the previous results by using the concavification method. This confirms the benefit of sending encoded data bits even if the decoding process is not supervised, e.g. when the decoder is an autonomous device. Our solution has two interesting features: it might be optimal not to use all channel resources; the informational content impacts the encoding process, since utility functions capture preferences on source symbols.

Information Theory
The More Unclear the Hearing, the Faster the Feeling: Speech Comprehension Changes Subjective Speech Rate

Liangjie Chen,Yangping Jin,Zhongshu Ge,Lingxi Lu,Liang Li

DOI: https://doi.org/10.1101/2023.10.17.562834

2023-01-01

Abstract:Imagine being in a foreign country, surrounded by locals speaking a language entirely alien to you. Does their conversation seem to race like a machine gun? The perception of speech rate, as a facet of time perception, greatly impacts our daily lives. However, the factors influencing this perception remain poorly understood. Through a series of three experiments (with sample sizes of 31, 27, and 25 respectively), we discovered that speech comprehension significantly alters listeners’ perception of speech rate. As comprehensibility diminishes, listeners tend to overestimate the speech rate. Intriguingly, this effect is further influenced by the listener’s language proficiency. Specifically, the better the listener’s language proficiency, the more likely their perception of speech rate is influenced by the speech comprehension. Our study corroborates the notion that biases in time perception may be prevalent in our everyday communication. These subjective experiences could prompt listeners to modify their behavior or processing strategies. ### Competing Interest Statement The authors have declared no competing interest.
Efficient human-like semantic representations via the Information Bottleneck principle

Noga Zaslavsky,Charles Kemp,Terry Regier,Naftali Tishby

DOI: https://doi.org/10.48550/arXiv.1808.03353

2018-08-09

Computation and Language

Abstract:Maintaining efficient semantic representations of the environment is a major challenge both for humans and for machines. While human languages represent useful solutions to this problem, it is not yet clear what computational principle could give rise to similar solutions in machines. In this work we propose an answer to this open question. We suggest that languages compress percepts into words by optimizing the Information Bottleneck (IB) tradeoff between the complexity and accuracy of their lexicons. We present empirical evidence that this principle may give rise to human-like semantic representations, by exploring how human languages categorize colors. We show that color naming systems across languages are near-optimal in the IB sense, and that these natural systems are similar to artificial IB color naming systems with a single tradeoff parameter controlling the cross-language variability. In addition, the IB systems evolve through a sequence of structural phase transitions, demonstrating a possible adaptation process. This work thus identifies a computational principle that characterizes human semantic systems, and that could usefully inform semantic representations in machines.

Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche

Human languages with greater information density have higher communication speed but lower conversation breadth

Human languages order information efficiently

Compression and the origins of Zipf's law of abbreviation

How Efficiency Shapes Human Language

Not lost in translation: neural responses shared across languages

Evolution of Efficient Symbolic Communication Codes

Discrete Messages Improve Communication Efficiency among Isolated Intelligent Agents

Evaluating performance of neural codes in model neural communication networks

A surprisal--duration trade-off across and within the world's languages

Zipf’s Law: Balancing Signal Usage Cost and Communication Efficiency

Information content versus word length in natural language: A reply to Ferrer-i-Cancho and Moscoso del Prado Martin [<a class="link-https" data-arxiv-id="1209.1751" href="https://arxiv.org/abs/1209.1751">arXiv:1209.1751</a>]

Compression as a universal principle of animal behavior

How Far can Neural Correlations Reduce Uncertainty? Comparison of Information Transmission Rates for Markov and Bernoulli Processes

From "um" to "yeah": Producing, predicting, and regulating information flow in human conversation

Information density as a predictor of communication dynamics

Revisiting Entropy Rate Constancy in Text

Towards Human-Agent Communication via the Information Bottleneck Principle

Information-Theoretic Limits of Strategic Communication

The More Unclear the Hearing, the Faster the Feeling: Speech Comprehension Changes Subjective Speech Rate

Efficient human-like semantic representations via the Information Bottleneck principle