RoboStory : A Tabletop Mixed Reality Framework for Children ’ s Role Play Storytelling

Haipeng Mi

2010-01-01

Abstract:In this paper, we present RoboStory, a mixed reality tabletop storytelling supporting system which enables children to create and present their own story in collaboration. A robot which can interact with characters, items and other children is used to enhance story presentation.

What problem does this paper attempt to address?

RoboTable: an Infrastructure for Intuitive Interaction with Mobile Robots in a Mixed-Reality Environment.

Haipeng Mi,Aleksander Krzywinski,Tomoki Fujita,Masanori Sugimoto

DOI: https://doi.org/10.1155/2012/301608

2012-01-01

Advances in Human-Computer Interaction

Abstract:This paper presents the design, development, and testing of a tabletop interface called RoboTable, which is an infrastructure supporting intuitive interaction with both mobile robots and virtual components in a mixed-reality environment. With a flexible software toolkit and specifically developed robots, the platform enables various modes of interaction with mobile robots. Using this platform, prototype applications are developed for two different application domains: RoboPong investigates the efficiency of the RoboTable system in game applications, and ExploreRobot explores the possibility of using robots and intuitive interaction to enhance learning.
RoboTable: a tabletop framework for tangible interaction with robots in a mixed reality.

Aleksander Krzywinski,Haipeng Mi,Weiqin Chen,Masanori Sugimoto

DOI: https://doi.org/10.1145/1690388.1690407

2009-01-01

Abstract:ABSTRACTCombining table top and tangible user interfaces is a relatively new research area. Many problems remain to be solved before users can benefit from tangible interactive table top systems. This paper presents our effort in this direction. RoboTable is an interactive table top system that enables users to naturally and intuitively manipulate robots. The goal of this research is to develop a software framework for human-robot interaction which combines table top, tangible objects, artificial intelligence and physics simulations and demonstrate the framework with game applications.
Exploitation Of Novel Multiplayer Gesture-Based Interaction And Virtual Puppetry For Digital Storytelling To Develop Children'S Narrative Skills

Hui Liang,Jian Chang,Shujie Deng,Can Chen,Ruofeng Tong,Jianjun Zhang

DOI: https://doi.org/10.1145/2817675.2817680

2015-01-01

Abstract:In recent years, digital storytelling has demonstrated powerful pedagogical functions by improving creativity, collaboration and intimacy among young children. Saturated with digital media technologies in their daily lives, the young generation demands natural interactive learning environments which offer multimodalities of feedback and meaningful immersive learning experiences. Virtual puppetry assisted storytelling system for young children, which utilises depth motion sensing technology and gesture control as the Human-Computer Interaction (HCI) method, has been proved to provide natural interactive learning experience for single player. In this paper, we designed and developed a novel system that allows multiple players to narrate, and most importantly, to interact with other characters and interactive virtual items in the virtual environment. We have conducted one user experiment with four young children for pedagogical evaluation and another user experiment with five postgraduate students for system evaluation. Our user study shows this novel digital storytelling system has great potential to stimulate learning abilities of young children through collaboration tasks.
Exploitation of Multiplayer Interaction and Development of Virtual Puppetry Storytelling Using Gesture Control and Stereoscopic Devices.

Hui Liang,Jian Chang,Shujie Deng,Can Chen,Ruofeng Tong,Jian Jun Zhang

DOI: https://doi.org/10.1002/cav.1727

IF: 1.01

2016-01-01

Computer Animation and Virtual Worlds

Abstract:With the rapid development of human–computer interaction technologies, the new media generation demands novel learning experiences with natural interaction and immersive experience. Considering that digital storytelling is a powerful pedagogical tool for young children, in this paper, we design an immersive storytelling environment that allows multiple players to use naturally interactive hand gestures to manipulate virtual puppetry for assisting narration. A set of multimodal interaction techniques is presented for a hybrid user interface that integrates existing 3D visualization and interaction devices including head‐mounted displays and depth motion sensor. In this system, the young players could intuitively use hand gestures to manipulate virtual puppets to perform a story and interact with props in a virtual stereoscopic environment. We have conducted a user experiment with four young children for pedagogical evaluation, as well as system acceptability and interactivity evaluation by postgraduate students. The results show that our framework has great potential to stimulate learning abilities of young children through collaboration tasks. The stereoscopic head‐mounted display outperformed the traditional monoscopic display in a comparison between the two.
Children's storytelling and programming with robotic characters

Kimiko Ryokai,Michael Jongseon Lee,Jonathan Micah Breitbart

DOI: https://doi.org/10.1145/1640233.1640240

2009-01-01

Abstract:We introduce mixed physical and digital authoring environments for children, which invite them to create stories with enriched drawings that are programmed to control robotic characters. These characters respond to the children's drawings as well as to their touch. Children create their stories by drawing props and programming how the robotic character should respond to those props and to physical touch. By drawing, programming the robotic character's behaviors, and organizing and negotiating the order and meanings of the props, children's story events unfold in creative ways. We present our iterative design process of developing and evaluating our prototypes with children. We discuss the role technology can play in supporting children's everyday creative storytelling.
A Humanoid Robot for Table Tennis Playing

Rong Xiong,Yong Liu,Hongbo Zheng

DOI: https://doi.org/10.1109/arso.2011.6301960

2011-01-01

Abstract:Humanoid robot has been one of the most active research topics in the field of robotics. Their human-like form and configuration gives it advantages in working in human-interactive environment. The bipedal walking capability makes them possible to step over and onto obstacles, providing accessibility and mobility in cluttered space. The multi-DOF design of arms and legs enables them assist or replace humans in their normal tasks, making human life easier and safer. Humanoid robots, with their human-like outlook, also bring better interactive experience and are expect to play a part in people's daily life and help the elderly and the children.
How Children Interpret Robots’ Contextual Behaviors in Live Theatre: Gaining Insights for Multi-Robot Theatre Design

Yu Peng,Yuan-Ling Feng,Nan Wang,Haipeng Mi

DOI: https://doi.org/10.1109/ro-man47096.2020.9223560

2020-01-01

Abstract:Robot theatre has been gradually used in children’s storytelling activities. The robotic character collaboratively exchange stories with children, react to their actions, and keep them engaged. However, the only robotic protagonist limits the richness of content and cannot fully support children’s creativity. To explore a tool with multiple robotic agents to better enhance story creation activities, we designed a simulated multi-robot theatre with three similarly shaped robots. We invited 28 kids (6-12 years old) to watch the video and retell the story. We encode children’s story recall and outline that robots’ contextual behaviors with emotion expression are reasonably enough for children over six-years-old to understand their performance. We also found no noticeable difference in story understanding among participants. We discuss the insights for designing robots’ contextual behaviors for story presentation and the considerations for implementing an accessible and feasible robot theatre creation platform.
Constructive Play: Designing for Role Play Stories with Interactive Play Objects.

Guanyun Wang,Ye Tao,Enmao Liu,Yunfan Wang,Cheng Yao,Fangtian Ying

DOI: https://doi.org/10.1145/2677199.2687907

2015-01-01

Abstract:ABSTRACTObjects, whether toys or not, are likely to become the center of children's game, which can trigger children's exploration and imagination, and build the world's environmental and psychological statements by manipulating something, to enhance the autonomy of children and inspire their creativity. This paper, based on scaffolding learning theory, puts forward the learning principle of constructive play suitable for learning in play school, suitable for cognitive behaviors of children aged 5 to 8 years old, and aims to encourage children to think creatively in the course of play, rather than learning knowledge and solve problems. We redesign the images, most commonly used by children RPG storytelling, and embed interactive intelligence, to better help children play games, stimulate their imagination and creativity in storytelling, and encourage them share with their companions and parents.
RoboTable2: a novel programming environment using physical robots on a tabletop platform

Masanori Sugimoto,Tomoki Fujita,Haipeng Mi,Aleksander Krzywinski

DOI: https://doi.org/10.1145/2071423.2071436

2011-01-01

Abstract:In this paper, we propose a novel environment called RoboTable2 that supports users with limited programming knowledge or experience in conducting robot programming. We have devised a tabletop platform that can simultaneously recognize multi-touch input and track physical robots, enabling users to conduct robot-programming tasks in an intuitive manner. A user study comparing the proposed and a conventional graphical programming environment was conducted, involving ten university students with no programming experience. The effects of the proposed environment were clarified via video analysis, questionnaires and usage logs. A pilot study was also conducted to verify how easily users could design and develop applications on RoboTable2. The lessons learned with respect to design guidelines for the proposed programming environment and issues for investigation are discussed.
An interactive 3D exploration narrative interface for storytelling.

ZhiYing Zhou,Adrian David Cheok,JiunHorng Pan,Yu Li

DOI: https://doi.org/10.1145/1017833.1017867

2004-01-01

Abstract:Storytelling is an effective and important educational means for children. With the augmented reality (AR) technology, storytelling becomes more and more interactive and intuitive in the sense of human computer interaction. Although AR technology is not new, it's potential in education is just beginning to be explored. In this paper, we present a 3D mixed media story cube which uses a foldable cube as the tangible and interactive storytelling interface. Here, we embed both the concept of AR and the concept of tangible interaction. Multiple modalities including speech, 3D audio, 3D graphics and touch are used to provide the user (especially children) with multi-sensory experiences in the process of storytelling. Our research explores a new interface for children education.
Sparkybot:An Embodied AI Agent-Powered Robot with Customizable Characters Andinteraction Behavior for Children

Yijie Guo,Zhenhan Huang,Ruhan Wang,Chih-Heng Li,Ruoyu Wu,Qirui Sun,Zhihao Yao,Haipeng Mi,Yu Peng

DOI: https://doi.org/10.1145/3586182.3615804

2023-01-01

Abstract:Sparkybot is a novel robotic platform designed to enrich children-robot interaction by leveraging embodied agent technology. The platform allows children to customize the interaction characters and behaviors of the robot, providing a more personalized and engaging experience. Two scenarios are demonstrated to showcase how children can customize and interact with the robot. Our goal is to provide a platform that enables children to interact with robots in a more creative and engaging way.
Robofriend: An Adpative Storytelling Robotic Teddy Bear -- Technical Report

Ido Glanz,Matan Weksler,Erez Karpas,Tzipi Horowitz-Kraus

DOI: https://doi.org/10.48550/arXiv.2301.01576

IF: 3.7

2023-01-04

Robotics

Abstract:In this paper we describe Robofriend, a robotic teddy bear for telling stories to young children. Robofriend adapts its behavior to keep the childrens' attention using reinforcement learning.
Roopkotha: A Companion Robot for Enhancing Interactive Storytelling with Natural Interaction

Kazi Mayesha Mehzabin,Md. Zahidul Islam,Md. Ashaduzzaman Nur,Mohammad Shidujaman,Hooman Samani,Haipeng Mi

DOI: https://doi.org/10.1109/iciprob62548.2024.10543303

2024-01-01

Abstract:Roopkotha is a storytelling robot that seamlessly combines traditional storytelling methods and technology, creating a captivating robot storyteller. We are creating a special prototype in the world of robots that tells stories in a way that is easy to understand and enjoy. In this era of technological advancement, Roopkotha combines voice recognition with Bangla Language processing, emotion recognition, human behavior detection. Roopkotha aims to revolutionize the way stories are told and engage with users on a deep emotional level. Furthermore, Roopkotha is equipped with advanced facial expressions and emotion recognition technology. The emotion recognition feature helps the robot to have a profound connection with the users.
Design and Analysis of a Collaborative Story Generation Game for Social Robots

Eric Nichols,Leo Gao,Yurii Vasylkiv,Randy Gomez

DOI: https://doi.org/10.3389/fcomp.2021.674333

2021-10-25

Frontiers in Computer Science

Abstract:Storytelling plays a central role in human socializing and entertainment, and research on conducting storytelling with robots is gaining interest. However, much of this research assumes that story content is curated. In this paper, we introduce the task of collaborative story generation , where an artificial intelligence agent, or a robot, and a person collaborate to create a unique story by taking turns adding to it. We present a collaborative story generation system which works with a human storyteller to create a story by generating new utterances based on the story so far. Our collaborative story generation system consists of a publicly-available large scale language model that was tuned on a dataset of writing prompts and short stories, and a ranker that samples from the language model and chooses the best possible output. We improve storytelling quality by optimizing the ranker’s sample size to strike a balance between quality and computational cost. Since latency can be detrimental to human-robot interaction, we examine the performance-latency trade-offs of our approach and find the optimal ranker sample size that strikes the best balance between quality and computational cost. We evaluate our system by having human participants play the collaborative story generation game and comparing the stories they create with our system to a naive baseline. Next, we conduct a detailed elicitation survey that sheds light on issues to consider when adapting our collaborative story generation system to a social robot. Finally, in a first step towards allowing human players to control the genre or mood of stories generated, we present preliminary work on steering story generation sentiment polarity with a sentiment analysis model. We find that our proposed method achieves a good balance of steering capability and text coherence. Our evaluation shows that participants have a positive view of collaborative story generation with a social robot and consider rich, emotive capabilities to be key to an enjoyable experience.
Mixed-Reality Robotic Games: Design Guidelines for Effective Entertainment With Consumer Robots

Filippo Gabriele Prattic,Fabrizio Lamberti,Filippo Gabriele Prattico

DOI: https://doi.org/10.1109/mce.2020.2988578

2021-01-01

IEEE Consumer Electronics Magazine

Abstract:In recent years, there has been an increasing interest in the use of robotic technology at home. A number of service robots appeared on the market, supporting customers in the execution of everyday tasks. Roughly at the same time, consumer-level robots started to be used also as toys or gaming companions. However, gaming possibilities provided by current off-the-shelf robotic products are generally quite limited, and this fact makes them quickly loose their attractiveness. A way that has been proven capable to boost robotic gaming and related devices consists in creating playful experiences in which physical and digital elements are combined together using mixed-reality technologies. However, these games differ significantly from digital- or physical-only experiences, and new design principles are required to support developers in their creative work. This article addresses such need, by drafting a set of guidelines which summarize developments carried out by the research community and their findings.

telecommunications,engineering, electrical & electronic,computer science, hardware & architecture
The Interplay of Robot Language Level with Children's Language Learning during Storytelling

Jacqueline Kory Westlund,Cynthia Breazeal

DOI: https://doi.org/10.1145/2701973.2701989

2015-03-02

Abstract:Children's oral language skills in preschool can predict their success in reading, writing, and academics in later schooling. Helping children improve their language skills early on could lead to more children succeeding later. As such, we examined the potential of a sociable robotic learning/teaching companion to support children's early language development. In a microgenetic study, 17 children played a storytelling game with the robot eight times over a two-month period. We evaluated whether a robot that "leveled" its stories to match the child's current abilities would lead to greater learning and language improvements than a robot that was not matched. All children learned new words, created stories, and enjoyed playing. Children who played with a matched robot used more words, and more diverse words, in their stories than unmatched children. Understanding the interplay between the robot's and the children's language will inform future work on robot companions that support children's education through play.
AI Stories: An Interactive Narrative System for Children

Ben Burtenshaw

DOI: https://doi.org/10.48550/arXiv.2011.04242

2020-11-09

Abstract:AI Stories is a proposed interactive dialogue system, that lets children co-create narrative worlds through conversation. Over the next three years this system will be developed and tested within pediatric wards, where it offers a useful resource between the gap of education and play. Telling and making stories is a fundamental part of language play, and its chatty and nonsensical qualities are important; therefore, the prologued usage an automated system offers is a benefit to children. In this paper I will present the current state of this project, in its more experimental and general guise. Conceptually story-telling through dialogue relates to the preprint interpretation of story, beyond the static and linear medium, where stories were performative, temporal, and social.

Computation and Language,Artificial Intelligence
Designing a realistic peer-like embodied conversational agent for supporting children's storytelling

Zhixin Li,Ying Xu

DOI: https://doi.org/10.48550/arXiv.2304.09399

2023-09-02

Abstract:Advances in artificial intelligence have facilitated the use of large language models (LLMs) and AI-generated synthetic media in education, which may inspire HCI researchers to develop technologies, in particular, embodied conversational agents (ECAs) to simulate the kind of scaffolding children might receive from a human partner. In this paper, we will propose a design prototype of a peer-like ECA named STARie that integrates multiple AI models - GPT-3, Speech Synthesis (Real-time Voice Cloning), VOCA (Voice Operated Character Animation), and FLAME (Faces Learned with an Articulated Model and Expressions) that aims to support narrative production in collaborative storytelling, specifically for children aged 4-8. However, designing a child-centered ECA raises concerns about age appropriateness, children privacy, gender choices of ECAs, and the uncanny valley effect. Thus, this paper will also discuss considerations and ethical concerns that must be taken into account when designing such an ECA. This proposal offers insights into the potential use of AI-generated synthetic media in child-centered AI design and how peer-like AI embodiment may support children\textquotesingle s storytelling.

Human-Computer Interaction
OriBot: a novel origami robot creation system to support children's STEAM learning

Yan Shi,Lijuan Liu,Xiaolong Lou,Yiwen Lu,Pan Zhang,Enmao Liu

DOI: https://doi.org/10.1007/s11042-024-20438-x

IF: 2.577

2024-11-23

Multimedia Tools and Applications

Abstract:Origami robots, as novel educational tools and toys, have substantial potential for broad application in children's education. Despite the recent advent of computational origami in the realms of human-computer interaction (HCI) and digital fabrication, children find it challenging to implement design and fabrication processes that leverage computational origami theory. Recognizing this challenge, we developed OriBot—an innovative system for creating origami robots, designed specifically for children aged 5 to 8. This comprehensive system comprises a hardware toolkit and a child-friendly software platform. The software is engineered to guide children through the processes of designing, fabricating, and operating origami robots. We evaluated the efficacy and user experience of the OriBot system by testing it with a sample group of 68 children. The results demonstrated that OriBot effectively facilitates children in the creation of origami robots, augmenting their understanding of robotics. Furthermore, it fosters positive interactions between children and robots while enhancing their engagement and interest in the activity all throughout.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Mixed-Reality Robot Behavior Replay: A System Implementation

Zhao Han,Tom Williams,Holly A. Yanco

DOI: https://doi.org/10.48550/arXiv.2210.00075

IF: 3.7

2022-09-30

Robotics

Abstract:As robots become increasingly complex, they must explain their behaviors to gain trust and acceptance. However, it may be difficult through verbal explanation alone to fully convey information about past behavior, especially regarding objects no longer present due to robots' or humans' actions. Humans often try to physically mimic past movements to accompany verbal explanations. Inspired by this human-human interaction, we describe the technical implementation of a system for past behavior replay for robots in this tool paper. Specifically, we used Behavior Trees to encode and separate robot behaviors, and schemaless MongoDB to structurally store and query the underlying sensor data and joint control messages for future replay. Our approach generalizes to different types of replays, including both manipulation and navigation replay, and visual (i.e., augmented reality (AR)) and auditory replay. Additionally, we briefly summarize a user study to further provide empirical evidence of its effectiveness and efficiency. Sample code and instructions are available on GitHub at https://github.com/umhan35/robot-behavior-replay.

RoboStory : A Tabletop Mixed Reality Framework for Children ’ s Role Play Storytelling

RoboTable: an Infrastructure for Intuitive Interaction with Mobile Robots in a Mixed-Reality Environment.

RoboTable: a tabletop framework for tangible interaction with robots in a mixed reality.

Exploitation Of Novel Multiplayer Gesture-Based Interaction And Virtual Puppetry For Digital Storytelling To Develop Children'S Narrative Skills

Exploitation of Multiplayer Interaction and Development of Virtual Puppetry Storytelling Using Gesture Control and Stereoscopic Devices.

Children's storytelling and programming with robotic characters

A Humanoid Robot for Table Tennis Playing

How Children Interpret Robots’ Contextual Behaviors in Live Theatre: Gaining Insights for Multi-Robot Theatre Design

Constructive Play: Designing for Role Play Stories with Interactive Play Objects.

RoboTable2: a novel programming environment using physical robots on a tabletop platform

An interactive 3D exploration narrative interface for storytelling.

Sparkybot:An Embodied AI Agent-Powered Robot with Customizable Characters Andinteraction Behavior for Children

Robofriend: An Adpative Storytelling Robotic Teddy Bear -- Technical Report

Roopkotha: A Companion Robot for Enhancing Interactive Storytelling with Natural Interaction

Design and Analysis of a Collaborative Story Generation Game for Social Robots

Mixed-Reality Robotic Games: Design Guidelines for Effective Entertainment With Consumer Robots

The Interplay of Robot Language Level with Children's Language Learning during Storytelling

AI Stories: An Interactive Narrative System for Children

Designing a realistic peer-like embodied conversational agent for supporting children's storytelling

OriBot: a novel origami robot creation system to support children's STEAM learning

Mixed-Reality Robot Behavior Replay: A System Implementation