Abstract:With the recent advances of open-domain story generation, the lack of reliable automatic evaluation metrics becomes an increasingly imperative issue that hinders the fast development of story generation. According to conducted researches in this regard, learnable evaluation metrics have promised more accurate assessments by having higher correlations with human judgments. A critical bottleneck of obtaining a reliable learnable evaluation metric is the lack of high-quality training data for classifiers to efficiently distinguish plausible and implausible machine-generated stories. Previous works relied on \textit{heuristically manipulated} plausible examples to mimic possible system drawbacks such as repetition, contradiction, or irrelevant content in the text level, which can be \textit{unnatural} and \textit{oversimplify} the characteristics of implausible machine-generated stories. We propose to tackle these issues by generating a more comprehensive set of implausible stories using {\em plots}, which are structured representations of controllable factors used to generate stories. Since these plots are compact and structured, it is easier to manipulate them to generate text with targeted undesirable properties, while at the same time maintain the grammatical correctness and naturalness of the generated sentences. To improve the quality of generated implausible stories, we further apply the adversarial filtering procedure presented by \citet{zellers2018swag} to select a more nuanced set of implausible texts. Experiments show that the evaluation metrics trained on our generated data result in more reliable automatic assessments that correlate remarkably better with human judgments compared to the baselines.

Controllable Neural Story Plot Generation via Reward Shaping

Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning

Neural Story Planning

Plot Writing From Pre-Trained Language Models

End-to-end Story Plot Generator

Learning to Generate Explainable Plots for Neural Story Generation

Learning to Predict Explainable Plots for Neural Story Generation

Story Realization: Expanding Plot Events into Sentences

From Plots to Endings: A Reinforced Pointer Generator for Story Ending Generation

PlotThread: Creating Expressive Storyline Visualizations using Reinforcement Learning

Guiding Neural Story Generation with Reader Models

PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking

Conveying the Predicted Future to Users: A Case Study of Story Plot Prediction

A Character-Centric Neural Model For Automated Story Generation

Story Shaping: Teaching Agents Human-like Behavior with Stories

CHAE: Fine-Grained Controllable Story Generation with Characters, Actions and Emotions

Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation

Plan, Write, and Revise: an Interactive System for Open-Domain Story Generation

Text2Reward: Reward Shaping with Language Models for Reinforcement Learning

What Makes A Good Story? Designing Composite Rewards for Visual Storytelling

Future Sight: Dynamic Story Generation with Large Pretrained Language Models