Abstract:Zipf's law is the main regularity of quantitative linguistics. Despite of many works devoted to foundations of this law, it is still unclear whether it is only a statistical regularity, or it has deeper relations with information-carrying structures of the text. This question relates to that of distinguishing a meaningful text (written in an unknown system) from a meaningless set of symbols that mimics statistical features of a text. Here we contribute to resolving these questions by comparing features of the first half of a text (from the beginning to the middle) to its second half. This comparison can uncover hidden effects, because the halves have the same values of many parameters (style, genre, author's vocabulary {\it etc}). In all studied texts we saw that for the first half Zipf's law applies from smaller ranks than in the second half, i.e. the law applies better to the first half. Also, words that hold Zipf's law in the first half are distributed more homogeneously over the text. These features do allow to distinguish a meaningful text from a random sequence of words. Our findings correlate with a number of textual characteristics that hold in most cases we studied: the first half is lexically richer, has longer and less repetitive words, more and shorter sentences, more punctuation signs and more paragraphs. These differences between the halves indicate on a higher hierarchic level of text organization that so far went unnoticed in text linguistics. They relate the validity of Zipf's law to textual information. A complete description of this effect requires new models, though one existing model can account for some of its aspects.

Maximal nonsymmetric entropy leads naturally to Zipf's law

Nonsymmetric entropy I: basic concepts and results

Zipf's law, power laws, and maximum entropy

Zipf'S Law Leads to Heaps' Law: Analyzing Their Relation in Finite-Size Systems

The Rank-Size Scaling Law and Entropy-Maximizing Principle

Maximal Diversity and Zipf's Law

Entropy of a Zipfian Distributed Lexicon

Big Data and Large Numbers. Interpreting Zipf's Law

Universal emergence of local Zipf's law

Maximum entropy principle for Renyi's and Tsallis' entropies

Maximal inequalities and a law of the iterated logarithm for negatively associated random fields

Universality of Zipf's Law

Relating Zipf's law to textual information

Zipf's Law for Atlas Models

Dynamical approach to Zipf's law

Zipf's law and maximum sustainable growth

Optimal coding and the origins of Zipfian laws

Log-log Convexity of Type-Token Growth in Zipf's Systems

The common patterns of abundance: the log series and Zipf's law

Entropy Sequences and Maximal Entropy Sets

Zipf's law, 1/f noise, and fractal hierarchy