A comment on the entropy of the Chinese language

R. Poon,K. Wong

DOI: https://doi.org/10.1109/TASSP.1976.1162865

1976-12-01

Abstract:Some statistical properties of the Chinese language are studied from an information-theoretic point of view. The entropies of Chinese characters, radicals, and Mandarin syllables are evaluated.

Computer Science,Linguistics

What problem does this paper attempt to address?

Quantitative Entropy Study of Language Complexity

R. R. Xie,W. B. Deng,D. J. Wang,L. P. Csernai,R.R. Xie,W.B. Deng,D.J. Wang,L.P. Csernai

DOI: https://doi.org/10.48550/arXiv.1611.04841

2016-11-14

Computation and Language

Abstract:We study the entropy of Chinese and English texts, based on characters in case of Chinese texts and based on words for both languages. Significant differences are found between the languages and between different personal styles of debating partners. The entropy analysis points in the direction of lower entropy, that is of higher complexity. Such a text analysis would be applied for individuals of different styles, a single individual at different age, as well as different groups of the population.
Entropy in Different Text Types.

Ruina Chen,Haitao Liu,Gabriel Altmann

DOI: https://doi.org/10.1093/llc/fqw008

IF: 1.299

2016-01-01

Digital Scholarship in the Humanities

Abstract:The present investigation is an attempt to investigate how the unique linguistic profile of different text types can be reflected in their respective entropy characteristics. With samples from the Lancaster Corpus of Mandarin Chinese and the Freiburg-Brown corpus of American English, the research investigates entropy performances in two dimensions: the relative entropy of words and their partof- speech (POS) on different sentential positions, and entropy of aspect markers. Our research yields the following results: First, it shows a strikingly similar distribution pattern in Chinese and English concerning the relative entropy of wordforms and POS-forms on different sentential positions. The relative entropy of word-forms in descending order yields: news > essays > official > academic > fiction, and the POS-forms yields: fiction > essays > news > academic > official. The relative entropy of POS-forms may be a more reliable indicator of syntactical differences, which helps to distinguish dichotomous 'narrative vs. expository' text types in both Chinese and English. Second, there exists a cross-linguistic difference concerning entropy of aspect markers, namely, Chinese displays higher relative entropy than English. This indicates that aspect-marking in terms of variation is more prominent in Chinese grammar than in English. The 'narrative vs. expository distinction' is also identified by entropy of aspect markers in both Chinese and English, though more obviously in Chinese.
Comparing entropies within the chinese language

Benjamin K. Tsou,Tom B. Y. Lai,Ka-po Chow

DOI: https://doi.org/10.1007/978-3-540-30211-7_49

2005-01-01

Abstract:Using a large synchronous Chinese corpus, we show how word and character entropy variations exhibit interesting differences in terms of time and space for different Chinese speech communities. We find that word entropy values are affected by the quality of the segmentation process. We also note that word entropies can be affected by proper nouns, which is the most volatile segment of the stable lexicon of the language. Our word and character entropy results provide interesting comparison with the earlier results and the average joint character entropies (a.k.a. entropy rates) of Chinese up to order 20 provided by us indicate that the limits of the conditional character entropies of Chinese for the different speech communities should be about 1 (or less). This invites questions on whether early convergence of character entropies would also entail word entropy convergence.
Entropy of Chinese and the perplexity of the language models

Jun Wu,Zuoying Wang

1996-01-01

Tien Tzu Hsueh Pao/Acta Electronica Sinica

Abstract:A method of estimating an upper bound of the entropy of printed Chinese is presented. A bound of 5.17 bits/character for the entropy is obtained by computing the entropy of the sample of Chinese corpus. The perplexity of several language models, which is a quantitative measurement for the ability of language models, is discussed. A new method of approximating high scale language model by the lower ones is also presented.
Estimating the Minimum Entropy of Chinese and Japanese Languages.

FJ Ren,K Yen

DOI: https://doi.org/10.1142/s0219622005001702

2005-01-01

International Journal of Information Technology & Decision Making

Abstract:The study of minimum entropy of a natural language has been an interesting research subject. For English, great progress has been made, but few reports on other languages have been found in literature. Based on two hypotheses on the conservation of information quantity, we proposed a method which can be used to estimate the minimum entropy of characters in natural languages. With a large quantity of translation corpus, this method enables us to estimate the minimum entropy without calculating the probability. Besides, as the scale of translation corpus increases, the fluctuation of the ratio between character quantities in any two languages becomes negligible. In this paper, we apply this method to the study of two languages of a large character total — Japanese and Chinese.
Chinese Word Length Motif And Its Evolution

Heng Chen,Junying Liang

DOI: https://doi.org/10.1515/9783110476637-004

2017-01-01

Abstract:In this paper, we compare the word length motif in modern spoken Chinese and written Chinese, and attempt to make clear how it evolves in written Chinese. The synchronic studies show that the rank-frequency distributions can be fitted with power law function y=ax(b), and the length distributions can be fitted with the Hyper-Pascal function. As for the diachronic studies, evolutionary regularities are found. On the one hand, we found an increasing trend of parameter a and a decreasing trend of parameter b in the rank-frequency distribution function y=axb. On the other hand, although the motif length distributions of ancient Chinese cannot be fitted with the Hyper-Pascal function, the entropy analyses show that there is a tendency that the distributions are more concentrated on some certain motif patterns.
Probability Distribution of Interlingual Lexical Divergences in Chinese and English: (dao) and Said in Hongloumeng

Yu Fang,Haitao Liu

2015-01-01

Glottometrics

Abstract:Previous studies have indicated that divergence exists in translation, which influences the quality of translation; such divergence follows some regularity which can be modeled by a probability distribution. The present article chose one verb dao from a Chinese literary classic Hongloumeng, and its English translation said to determine whether the frequencies of the two verbs develop according to a diversification process. Furthermore, we tend to find differences between the two English versions of Hawkes and Yang Xianyi regarding dao's translation, and we also investigated the role of the three causes of divergence in the diversification process. The result indicates that both dao's translations and the original text of said is in good agreement with the modified right-truncated Zipf-Alekseev distribution. Three major reasons were found for causing differences between the two translations: the nature of language, translators' subjectivity, and context. These are also the three major reasons for the diversification and differences of the two translated versions.
Synergetic Properties of Chinese Verb Valency.

Song Gao,Hongxin Zhang,Haitao Liu

DOI: https://doi.org/10.1080/09296174.2013.856132

2013-01-01

Journal of Quantitative Linguistics

Abstract:This paper analyses the 500 most frequent verbs in contemporary Chinese and investigates their synergetic properties. The results show that the rank-frequency distributions of both valency and polysemy abide by a power-law distribution and that valency and polysemy of these verbs abide by the Good distribution and the positive negative binomial distribution respectively. Statistical analysis indicates that the greater valency a verb has, the greater polysemy, frequency and polytextuality can be expected. To find a suitable unit of measurement, strokes, a feature of the Chinese writing system, and pinyin, a phonemic transcription system of Chinese, are introduced in the present study to measure word length. These two types of word length correlate highly with each other, and both exhibit some inverse interrelations with other linguistic variables like verb valency, polysemy, frequency and polytextuality. Finally, based on these findings, we present a synergetic model (control cycle) of Chinese verb valency.
An Estimate Method Of The Minimum Entropy Of Natural Languages

Fj Ren,S Mitsuyoshi,K Yen,Cq Zong,Hb Zhu

DOI: https://doi.org/10.1007/3-540-36456-0_39

2003-01-01

Abstract:The study of minimum entropy of English has a long history and has made a great progress, but only a few studies on other languages have been reported in literature so far. In this paper, we present a new method to estimate the minimum entropy of character in natural languages, based on two hypotheses of conservation of information quantity. We also verified the hypotheses empirically through experiments with two natural languages, Japanese and Chinese.
Deviation of Zipf's and Heaps' Laws in Human Languages with Limited Dictionary Sizes

Linyuan Lü,Zi-Ke Zhang,Tao Zhou

DOI: https://doi.org/10.1038/srep01082

IF: 4.6

2013-01-01

Scientific Reports

Abstract:Zipf's law on word frequency and Heaps' law on the growth of distinct words are observed in Indo-European language family, but it does not hold for languages like Chinese, Japanese and Korean. These languages consist of characters and are of very limited dictionary sizes. Extensive experiments show that: (i) The character frequency distribution follows a power law with exponent close to one, at which the corresponding Zipf's exponent diverges. Indeed, the character frequency decays exponentially in the Zipf's plot. (ii) The number of distinct characters grows with the text length in three stages: It grows linearly in the beginning, then turns to a logarithmical form and eventually saturates. A theoretical model for writing process is proposed, which embodies the rich-get-richer mechanism and the effects of limited dictionary size. Experiments, simulations and analytical solutions agree well with each other. This work refines the understanding about Zipf's and Heaps' laws in human language systems.
Linguistic emergence from a networks approach: The case of modern Chinese two-character words.

Jin Cong,Haitao Liu

DOI: https://doi.org/10.1371/journal.pone.0259818

IF: 3.7

2021-01-01

PLoS ONE

Abstract:The models of linguistic networks and their analytical tools constitute a potential methodology for investigating the formation of structural patterns in actual language use. Research with this methodology has just started, which can hopefully shed light on the emergent nature of linguistic structure. This study attempts to employ linguistic networks to investigate the formation of modern Chinese two-character words (as structural units based on the chunking of their component characters) in the actual use of modern Chinese, which manifests itself as continuous streams of Chinese characters. Network models were constructed based on authentic Chinese language data, with Chinese characters as nodes, their co-occurrence relations as directed links, and the co-occurrence frequencies as link weights. Quantitative analysis of the network models has shown that a Chinese two-character word can highlight itself as a two-node island, i.e., a cohesive sub-network with its two component characters co-occurring more frequently than they co-occur with the other characters. This highlighting mechanism may play a vital role in the formation and acquisition of two-character words in actual language use. Moreover, this mechanism may also throw some light on the emergence of other structural phenomena (with the chunking of specific linguistic units as their basis).
On the Idiosyncrasies of the Mandarin Chinese Classifier System

Shijia Liu,Hongyuan Mei,Adina Williams,Ryan Cotterell

DOI: https://doi.org/10.48550/arXiv.1902.10193

2020-05-23

Abstract:While idiosyncrasies of the Chinese classifier system have been a richly studied topic among linguists (Adams and Conklin, 1973; Erbaugh, 1986; Lakoff, 1986), not much work has been done to quantify them with statistical methods. In this paper, we introduce an information-theoretic approach to measuring idiosyncrasy; we examine how much the uncertainty in Mandarin Chinese classifiers can be reduced by knowing semantic information about the nouns that the classifiers modify. Using the empirical distribution of classifiers from the parsed Chinese Gigaword corpus (Graff et al., 2005), we compute the mutual information (in bits) between the distribution over classifiers and distributions over other linguistic quantities. We investigate whether semantic classes of nouns and adjectives differ in how much they reduce uncertainty in classifier choice, and find that it is not fully idiosyncratic; while there are no obvious trends for the majority of semantic classes, shape nouns reduce uncertainty in classifier choice the most.

Computation and Language
A New Look at the Classical Entropy of Written English

Fabio G. Guerrero

DOI: https://doi.org/10.48550/arXiv.0911.2284

2009-11-19

Abstract:A simple method for finding the entropy and redundancy of a reasonable long sample of English text by direct computer processing and from first principles according to Shannon theory is presented. As an example, results on the entropy of the English language have been obtained based on a total of 20.3 million characters of written English, considering symbols from one to five hundred characters in length. Besides a more realistic value of the entropy of English, a new perspective on some classic entropy-related concepts is presented. This method can also be extended to other Latin languages. Some implications for practical applications such as plagiarism-detection software, and the minimum number of words that should be used in social Internet network messaging, are discussed.

Computation and Language
Maximum Entropy, Word-Frequency, Chinese Characters, and Multiple Meanings

Xiaoyong Yan,Petter Minnhagen

DOI: https://doi.org/10.1371/journal.pone.0125592

IF: 3.7

2015-05-08

PLoS ONE

Abstract:The word-frequency distribution of a text written by an author is well accounted for by a maximum entropy distribution, the RGF (random group formation)-prediction. The RGF-distribution is completely determined by the a priori values of the total number of words in the text (M), the number of distinct words (N) and the number of repetitions of the most common word (k(max)). It is here shown that this maximum entropy prediction also describes a text written in Chinese characters. In particular it is shown that although the same Chinese text written in words and Chinese characters have quite differently shaped distributions, they are nevertheless both well predicted by their respective three a priori characteristic values. It is pointed out that this is analogous to the change in the shape of the distribution when translating a given text to another language. Another consequence of the RGF-prediction is that taking a part of a long text will change the input parameters (M, N, k(max)) and consequently also the shape of the frequency distribution. This is explicitly confirmed for texts written in Chinese characters. Since the RGF-prediction has no system-specific information beyond the three a priori values (M, N, k(max)), any specific language characteristic has to be sought in systematic deviations from the RGF-prediction and the measured frequencies. One such systematic deviation is identified and, through a statistical information theoretical argument and an extended RGF-model, it is proposed that this deviation is caused by multiple meanings of Chinese characters. The effect is stronger for Chinese characters than for Chinese words. The relation between Zipf's law, the Simon-model for texts and the present results are discussed.

multidisciplinary sciences
Perplexity Measuring of Language Model and the Entropy Estimating of Chinese

ZHANG Yang-sen,CAO Yuan-da,YU Shi-wen

DOI: https://doi.org/10.3969/j.issn.1000-1220.2006.10.031

2006-01-01

Abstract:The perplexity measuring methods of language model are quantificationally expressed from entropy angle with information theory. We have result in the inference that the more little entropy estimated by a LM is, the more well this model's performance, as well as the new model obtained by combining linearly two (n-1)-gram statistical linguistic models, its performance is better than (n-1)-gram model, but is not as good as n-gram model.
EMPIRICAL CHARACTERIZATION OF MODERN CHINESE AS A MULTI-LEVEL SYSTEM FROM THE COMPLEX NETWORK APPROACH

Haitao Liu,Jin Cong

2014-01-01

Journal of Chinese Linguistics

Abstract:The conception of language as a multi-level system is widely accepted in modern linguistics. However, due to the absence of operational methodology, there has been no empirical characterization of any language as a multi-level system in the current body of linguistic research. The present study attempts to fill this gap by examining modern Chinese as a multi-level system from the complex network approach. Based on the same body of corpus data, four levels as language sub-systems along the meaning-form dimension of the language are modeled as linguistic networks. On the basis of topological analysis of the four linguistic networks, the organizational patterns of the four language sub-systems are described and compared quantitatively and the results are interpreted from an interdisciplinary perspective. To varying degrees, the four network models exhibit a series of non-trivial statistical patterns, which shed light on different aspects of the organization of the four sub-systems. The similarities and differences of the four sub-systems in organizational patterns reflect the relationships between them. In more than one way, the results of this study point to the close connection between relevant properties of the language and human cognition, which facilitates linguistic performance of different types.
Quantitative Analysis of Chinese and English Verb Valencies Based on Probabilistic Valency Pattern Theory

Haitao Liu

DOI: https://doi.org/10.1007/978-3-031-06547-7_12

2011-01-01

Journal of Quantitative Linguistics

Abstract:This study examines the similarities and differences between Chinese and English verb valencies based on the Probabilistic Valency Pattern Theory (PVPT). We adopted the Parallel Universal Dependencies treebanks of Chinese and English to ensure that the comparison is under the same semantic meanings conveyed. The results show that (1) The verb valencies of both languages share similar distributions. One important difference is that Chinese has significantly more monovalent verbs (valency equals one) than English does; (2) For conveying similar meanings, Chinese adopts more verbs than English does, while the average combinatorial ability of Chinese verbs is relatively smaller; (3) The overall probabilistic valency pattern (PVP) of verbs in Chinese and English are similar; however, those of specific high-frequent verbs in Chinese and English demonstrate their own features. The findings may shed light on depicting the characteristics of Chinese and English verbs, thus facilitating studies in both linguistics and natural language processing.
English and Chinese Languages As Weighted Complex Networks

Long Sheng,Chunguang Li

DOI: https://doi.org/10.1016/j.physa.2009.02.043

2009-01-01

Abstract:In this paper, we analyze statistical properties of English and Chinese written human language within the framework of weighted complex networks. The two language networks are based on an English novel and a Chinese biography, respectively, and both of the networks are constructed in the same way. By comparing the intensity and density of connections between the two networks, we find that high weight connections in Chinese language networks prevail more than those in English language networks. Furthermore, some of the topological and weighted quantities are compared. The results display some differences in the structural organizations between the two language networks. These observations indicate that the two languages may have different linguistic mechanisms and different combinatorial natures.
Exploring Colligation Diversity and Grammaticalization in Chinese: an Entropy-Based Approach

Biyan Yu,Yue Jiang

DOI: https://doi.org/10.1080/09296174.2024.2395072

2024-01-01

Journal of Quantitative Linguistics

Abstract:As the process of grammaticalization unfolds, it remains to be determined whether a word could co-occur with more words in contexts or would be restricted to fewer words. Based on the Lancaster Corpus of Mandarin Chinese (LCMC), this study examines the differences in colligation diversity between lexical and grammatical words in Chinese by using entropy, aiming to explore how the colligational behaviour of the left and right sides of Chinese words changes accordingly with increasing grammaticalization. The comparisons of colligation diversity between two sides and across word categories reveal that lexical words show quite similar levels of colligation diversity on the left, which makes them significantly different from grammatical words. More category-specific observations are disclosed by entropy-based approach. In the case of grammaticalization, an increase in entropy values denotes more types with a more uniform distribution, which is suggested to be the manifestation of semantic bleaching. Conversely, a decrease in entropy values may be an indicator of an increasing bondedness. The discussion on how grammaticalization affects the colligational behaviour of words should be based on the specific pathways of grammaticalization concerning word categories, as well as specific sides of words.
Entropy of a Zipfian Distributed Lexicon

Leonardo Carneiro Araujo,Thaïs Cristófaro-Silva,Hani Camille Yehia

DOI: https://doi.org/10.48550/arXiv.1307.0898

2013-07-03

Abstract:This article presents the calculation of the entropy of a system with Zipfian distribution and shows that a communication system tends to present an exponent value close to one, but still greater than one, so that it might maximize entropy and hold a feasible lexicon with an increasing size. This result is in agreement with what is observed in natural languages and with the balance between the speaker and listener communication efforts. On the other hand, the entropy of the communicating source is very sensitive to the exponent value as well as the length of the observable data, making it a poor parameter to characterize the communication process.

Applications,Physics and Society

A comment on the entropy of the Chinese language

Quantitative Entropy Study of Language Complexity

Entropy in Different Text Types.

Comparing entropies within the chinese language

Entropy of Chinese and the perplexity of the language models

Estimating the Minimum Entropy of Chinese and Japanese Languages.

Chinese Word Length Motif And Its Evolution

Probability Distribution of Interlingual Lexical Divergences in Chinese and English: (dao) and Said in Hongloumeng

Synergetic Properties of Chinese Verb Valency.

An Estimate Method Of The Minimum Entropy Of Natural Languages

Deviation of Zipf's and Heaps' Laws in Human Languages with Limited Dictionary Sizes

Linguistic emergence from a networks approach: The case of modern Chinese two-character words.

On the Idiosyncrasies of the Mandarin Chinese Classifier System

A New Look at the Classical Entropy of Written English

Maximum Entropy, Word-Frequency, Chinese Characters, and Multiple Meanings

Perplexity Measuring of Language Model and the Entropy Estimating of Chinese

EMPIRICAL CHARACTERIZATION OF MODERN CHINESE AS A MULTI-LEVEL SYSTEM FROM THE COMPLEX NETWORK APPROACH

Quantitative Analysis of Chinese and English Verb Valencies Based on Probabilistic Valency Pattern Theory

English and Chinese Languages As Weighted Complex Networks

Exploring Colligation Diversity and Grammaticalization in Chinese: an Entropy-Based Approach

Entropy of a Zipfian Distributed Lexicon