Abstract:The formation of sentences is a highly structured and history-dependent process. The probability of using a specific word in a sentence strongly depends on the 'history' of word-usage earlier in that sentence. We study a simple history-dependent model of text generation assuming that the sample-space of word usage reduces along sentence formation, on average. We first show that the model explains the approximate Zipf law found in word frequencies as a direct consequence of sample-space reduction. We then empirically quantify the amount of sample-space reduction in the sentences of ten famous English books, by analysis of corresponding word-transition tables that capture which words can follow any given word in a text. We find a highly nested structure in these transition tables and show that this `nestedness' is tightly related to the power law exponents of the observed word frequency distributions. With the proposed model it is possible to understand that the nestedness of a text can be the origin of the actual scaling exponent, and that deviations from the exact Zipf law can be understood by variations of the degree of nestedness on a book-by-book basis. On a theoretical level we are able to show that in case of weak nesting, Zipf's law breaks down in a fast transition. Unlike previous attempts to understand Zipf's law in language the sample-space reducing model is not based on assumptions of multiplicative, preferential, or self-organised critical mechanisms behind language formation, but simply used the empirically quantifiable parameter 'nestedness' to understand the statistics of word frequencies.

Proper Interpretation of Heaps' and Zipf's Laws

Zipf'S Law Leads to Heaps' Law: Analyzing Their Relation in Finite-Size Systems

Deviation of Zipf's and Heaps' Laws in Human Languages with Limited Dictionary Sizes

Scaling Laws in Human Language

Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models

A scaling law beyond Zipf's law and its relation to Heaps' law

On Zipf's Law in Korean Language

Zipf's Law in 50 Languages: Its Structural Pattern, Linguistic Interpretation, and Cognitive Motivation

Understanding Zipf's law of word frequencies through sample-space collapse in sentence formation

Log-log Convexity of Type-Token Growth in Zipf's Systems

Universality of Zipf's Law

Asymptotically normal estimators for Zipf's law

Big Data and Large Numbers. Interpreting Zipf's Law

Maximal Diversity and Zipf's Law

Zipf's and Taylor's Laws

The common patterns of abundance: the log series and Zipf's law

Co-occurrence of the Benford-like and Zipf Laws Arising from the Texts Representing Human and Artificial Languages

Large-Scale Analysis of Zipf’s Law in English Texts

A Simple Derivation of the Heap's Law from the Generalized Zipf's Law

Zipf's Law for Atlas Models

Rank-frequency Relation for Chinese Characters