What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation

Dingyi Yang,Qin Jin

2024-08-27

Abstract:With the development of artificial intelligence, particularly the success of Large Language Models (LLMs), the quantity and quality of automatically generated stories have significantly increased. This has led to the need for automatic story evaluation to assess the generative capabilities of computing systems and analyze the quality of both automatic-generated and human-written stories. Evaluating a story can be more challenging than other generation evaluation tasks. While tasks like machine translation primarily focus on assessing the aspects of fluency and accuracy, story evaluation demands complex additional measures such as overall coherence, character development, interestingness, etc. This requires a thorough review of relevant research. In this survey, we first summarize existing storytelling tasks, including text-to-text, visual-to-text, and text-to-visual. We highlight their evaluation challenges, identify various human criteria to measure stories, and present existing benchmark datasets. Then, we propose a taxonomy to organize evaluation metrics that have been developed or can be adopted for story evaluation. We also provide descriptions of these metrics, along with the discussion of their merits and limitations. Later, we discuss the human-AI collaboration for story evaluation and generation. Finally, we suggest potential future research directions, extending from story evaluation to general evaluations.

Computation and Language

What problem does this paper attempt to address?

The problem this paper attempts to address is the quality assessment of automatic story generation. With the development of large language models (LLMs), both the quantity and quality of automatically generated stories have significantly improved, which has led to a demand for automatic story evaluation. Automatic story evaluation not only needs to assess the generation capability of the systems but also analyze the quality of the plot in automatically generated stories compared to human-written ones. Compared to other generation tasks like machine translation, story evaluation is more complex as it involves multiple aspects such as overall coherence, character development, and entertainment value. Therefore, this review article aims to summarize existing story generation tasks, identify evaluation criteria, and propose a classification system to organize evaluation metrics for a better understanding and assessment of automatically generated stories.