Connection is All You Need: A Multimodal Human-AI Co-Creation Storytelling System to Support Children's Multi-Level Narrative Skills

Lyumanshan Ye,Jiandong Jiang,Yuhan Liu,Yihan Ran,Shuning Zhang,Yanpu Yin,Pengfei Liu,Danni Chang

2024-12-12

Abstract:Children develop narrative skills by understanding and actively building connections between elements, image-text matching and consequences. However, it is challenging for children to clearly grasp these multi-level links only through explanations of text or facilitator's speech. To address this, we developed Colin, an interactive storytelling tool that supports children's multi-level narrative skills through both voice and visual modalities. In the generation stage, Colin supports facilitator to define and review generated text and image content freely. In the understanding stage, a question-feedback model helps children understand multi-level connections while co-creating stories with Colin. In the building phase, Colin actively encourages children to create connections between elements through drawing and speaking. A user study with 20 participants evaluated Colin by measuring children's engagement, understanding of cause-and-effect relationships, and the quality of their new story creations. Our results demonstrated that Colin significantly enhances the development of children's narrative skills across multiple levels.

Human-Computer Interaction

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: Children have difficulties in understanding multi - level connections (such as the relationships between elements, text - image matching, and cause - and - effect relationships) through texts or explanations from guides. To meet this challenge, the research team has developed Colin, an interactive story - creation tool, aiming to support the development of children's multi - level narrative skills through voice and visual modalities. Specifically, Colin helps children in the following ways:

1. **Generation stage**: Support the guide to define and review the generated text and image content.
2. **Comprehension stage**: Help children understand multi - level connections through the question - feedback model while co - creating stories with Colin.
3. **Construction stage**: Actively encourage children to create connections between elements by drawing and speaking.

Through these functions, Colin aims to significantly improve the development of children's narrative skills at multiple levels. The research evaluated the effectiveness of Colin through user studies, including children's participation, understanding of cause - and - effect relationships, and the quality of new story creation. The results show that Colin performs excellently in promoting the development of children's narrative skills.