Abstract:We introduce Vocal Sandbox, a framework for enabling seamless human-robot collaboration in situated environments. Systems in our framework are characterized by their ability to adapt and continually learn at multiple levels of abstraction from diverse teaching modalities such as spoken dialogue, object keypoints, and kinesthetic demonstrations. To enable such adaptation, we design lightweight and interpretable learning algorithms that allow users to build an understanding and co-adapt to a robot's capabilities in real-time, as they teach new behaviors. For example, after demonstrating a new low-level skill for "tracking around" an object, users are provided with trajectory visualizations of the robot's intended motion when asked to track a new object. Similarly, users teach high-level planning behaviors through spoken dialogue, using pretrained language models to synthesize behaviors such as "packing an object away" as compositions of low-level skills $-$ concepts that can be reused and built upon. We evaluate Vocal Sandbox in two settings: collaborative gift bag assembly and LEGO stop-motion animation. In the first setting, we run systematic ablations and user studies with 8 non-expert participants, highlighting the impact of multi-level teaching. Across 23 hours of total robot interaction time, users teach 17 new high-level behaviors with an average of 16 novel low-level skills, requiring 22.1% less active supervision compared to baselines and yielding more complex autonomous performance (+19.7%) with fewer failures (-67.1%). Qualitatively, users strongly prefer Vocal Sandbox systems due to their ease of use (+20.6%) and overall performance (+13.9%). Finally, we pair an experienced system-user with a robot to film a stop-motion animation; over two hours of continuous collaboration, the user teaches progressively more complex motion skills to shoot a 52 second (232 frame) movie.

Robo Sapiens

Giving Robots a Voice: Human-in-the-Loop Voice Creation and open-ended Labeling

Natural Language for Human-Robot Collaboration: Problems Beyond Language Grounding

Talking with Robots: Opportunities and Challenges

Facilitating Human-Robot Collaboration through Natural Vocal Conversations

Robot‐by‐voice: experiments on commanding an industrial robot using the human voice

Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community

More than Chit-Chat: Developing Robots for Small-Talk Interactions

Read the Room: Adapting a Robot's Voice to Ambient and Social Contexts

Get It Right: Improving Comprehensibility with Adaptable Speech Expression of a Humanoid Service Robot

SIFToM: Robust Spoken Instruction Following through Theory of Mind

A Biomimetic Vocalisation System for MiRo

Whither the Priors for (Vocal) Interactivity?

Interactive Robot Learning of Gestures, Language and Affordances

Conversational AI for multi-agent communication in Natural Language

You just do not understand me! Speech Recognition in Human Robot Interaction

Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration

Building for Speech: Designing the Next Generation of Social Robots for Audio Interaction

Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction

Single-Channel Robot Ego-Speech Filtering during Human-Robot Interaction

Pro-active service robots in a health care framework: vocal interaction using natural language and prosody