Abstract:The development of large language models (LLMs) capable of following instructions and engaging in conversational interactions sparked increased interest in their utilization across various support tools. We investigate the utility of modern LLMs in assisting professional writers via an empirical user study (n=30). The design of our collaborative writing interface is grounded in the cognitive process model of writing that views writing as a goal-oriented thinking process encompassing non-linear cognitive activities: planning, translating, and reviewing. Participants are asked to submit a post-completion survey to provide feedback on the potential and pitfalls of LLMs as writing collaborators. Upon analyzing the writer-LLM interactions, we find that while writers seek LLM's help across all three types of cognitive activities, they find LLMs more helpful in translation and reviewing. Our findings from analyzing both the interactions and the survey responses highlight future research directions in creative writing assistance using LLMs.

What problem does this paper attempt to address?

The problem this paper attempts to address is: How to effectively support professional writers in the creative writing process using large language models (LLMs). Specifically, the authors explore the practical utility and potential issues of modern LLMs in assisting professional writers through an empirical study. ### Research Background In recent years, large language models (LLMs) have garnered widespread attention in various support tools due to their powerful generative capabilities and conversational interaction abilities. However, the effectiveness of these models in actual creative writing remains unclear, especially among professional writers. This paper aims to evaluate the practical utility of LLMs in the creative writing process through empirical research and to explore their performance in different cognitive activities. ### Research Objectives 1. **Design a collaborative writing interface based on cognitive process theory**: This interface allows professional writers to seek help from LLMs in three non-linear cognitive activities: planning, translating, and reviewing. 2. **Collect data**: Gather stories and human-computer interaction data through collaboration with 30 emerging writers. 3. **Analyze data**: Evaluate the practical utility and potential issues of LLMs in creative writing by analyzing the interaction data between writers and LLMs and feedback from questionnaires. ### Key Findings 1. **Writers seek help from LLMs in all cognitive activities**, but it is more useful in the translating and reviewing stages. 2. **Writers exhibit a non-linear writing process when using LLMs**, validating the effectiveness of the design choices. 3. **Current model limitations**: Including repetitiveness, over-reliance on clichés, lack of nuance, metaphor or symbolic meaning, and overly moralistic and predictable endings. ### Future Research Directions 1. **Improve model training**: To better meet the needs of professional writers, especially in handling dark themes and understanding writer intent. 2. **Further explore human-computer collaboration frameworks**: Design more flexible and efficient collaborative writing tools. ### Conclusion Through empirical research, this paper systematically evaluates for the first time the practical utility of modern LLMs in professional writers' creative writing and proposes future research directions. The research results not only provide a theoretical basis for the design of creative writing support tools but also offer practical guidance for further optimizing the application of LLMs in creative writing.

Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers

Evaluating Large Language Model Creativity from a Literary Perspective

On the Creativity of Large Language Models

Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking

Steering Large Language Models to Evaluate and Amplify Creativity

GhostWriter: Augmenting Collaborative Human-AI Writing Experiences Through Personalization and Agency

Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation

Art or Artifice? Large Language Models and the False Promise of Creativity

Assessing and Understanding Creativity in Large Language Models

Large Language Model in Creative Work: The Role of Collaboration Modality and User Expertise

Characterising the Creative Process in Humans and Large Language Models

LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play

"It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language Models

Divergent Creativity in Humans and Large Language Models

Inspo: Writing Stories with a Flock of AIs and Humans

AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation

Exploring the Potential of Large Language Models in Artistic Creation: Collaboration and Reflection on Creative Programming

Task Supportive and Personalized Human-Large Language Model Interaction: A User Study

ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models

Apprentices to Research Assistants: Advancing Research with Large Language Models

Enhancing Creativity in Large Language Models through Associative Thinking Strategies