Understanding Literary Texts by LLMs: A Case Study of Ancient Chinese Poetry

Cheng Zhao,Bin Wang,Zhen Wang

2024-09-11

Abstract:The birth and rapid development of large language models (LLMs) have caused quite a stir in the field of literature. Once considered unattainable, AI's role in literary creation is increasingly becoming a reality. In genres such as poetry, jokes, and short stories, numerous AI tools have emerged, offering refreshing new perspectives. However, it's difficult to further improve the quality of these works. This is primarily because understanding and appreciating a good literary work involves a considerable threshold, such as knowledge of literary theory, aesthetic sensibility, interdisciplinary knowledge. Therefore, authoritative data in this area is quite lacking. Additionally, evaluating literary works is often complex and hard to fully quantify, which directly hinders the further development of AI creation. To address this issue, this paper attempts to explore the mysteries of literary texts from the perspective of LLMs, using ancient Chinese poetry as an example for experimentation. First, we collected a variety of ancient poems from different sources and had experts annotate a small portion of them. Then, we designed a range of comprehension metrics based on LLMs to evaluate all these poems. Finally, we analyzed the correlations and differences between various poem collections to identify literary patterns. Through our experiments, we observed a series of enlightening phenomena that provide technical support for the future development of high-level literary creation based on LLMs.

Computation and Language

What problem does this paper attempt to address?

The main problem this paper attempts to address is the understanding and evaluation of classical literary works, particularly ancient Chinese poetry, through large language models (LLMs). Specifically, the paper aims to:

1. **Address the challenge of evaluating high-quality literary creation**: The current evaluation of AI-generated literary works primarily relies on human assessment. When the quality of the work reaches a certain level, non-expert evaluations become unreliable, while expert evaluations are difficult to scale. Therefore, the paper proposes a literature understanding framework based on LLMs to improve the efficiency and accuracy of literary work quality evaluation.

2. **Explore methods for understanding literary texts**: By collecting ancient poems from different sources and having experts annotate some of the poems, a series of LLM-based understanding metrics are designed to evaluate these poems. These metrics include, but are not limited to, lexical statistics, embedding vectors, hidden states, and output probabilities.

3. **Identify literary patterns**: By analyzing the metrics of different poetry collections, the paper identifies patterns and differences in literary works, thereby providing technical support for future advanced literary creation.

4. **Quantify the evaluation of literary works**: Using a small amount of expert-annotated data to address the challenges of evaluating literary texts, the paper demonstrates the high scalability of this method. It argues that with the support of LLMs, the evaluation of more types of literary works can be effectively quantified, promoting the prosperous development of AI literary creation.

In summary, through a series of experiments and technical means, this paper explores how to better understand and evaluate ancient Chinese poetry using LLMs, providing new ideas and technical support for the high-quality development of future literary creation.