Abstract:This paper addresses multimodal “scene understanding” for a robot using audio-visual and text information. Scene understanding is defined by extracting six-W information such as What, When, Where, Who, Why, and hoW on the surrounding environment. Although scene understanding for a robot has been studied in the fields of robot vision and audition, only the first four W s except for why and how information were considered. We, thus, focus on extracting how information, in particular, on cooking scenes. In cooking scenes, we define how information as a cooking procedure, and it is useful that a robot gives appropriate advice for cooking. To realize such cooking support, we propose a multimodal cooking procedure recognition framework consisting of Convolutional Neural Network (CNN), and Hierarchical Hidden Markov Model (HHMM). CNN is knows as one of the most advanced classifiers, and it is applied to recognize a cooking events from audio and visual information. HHMM models a cooking procedure represented by a sequence of cooking events, which is defined as a relationship between cooking events using text data obtained from web, and the cooking events classified with CNN. Therefore, our proposed framework integrates these three types of modalities. We constructed an interactive cooking support system based on the proposed framework, which advice a next step in the current cooking procedure through human-robot communication. Preliminary results with simulated and real recorded multi-modal scenes showed the robustness of the proposed framework in a noisy and/or occluded situation.

Real-world cooking robot system from recipes based on food state recognition using foundation models and PDDL

Real-World Cooking Robot System from Recipes Based on Food State Recognition Using Foundation Models and PDDL

Recognition of Heat-Induced Food State Changes by Time-Series Use of Vision-Language Model for Cooking Robot

Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-Box Optimization

Cook2LTL: Translating Cooking Recipes to LTL Formulae using Large Language Models

LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application

Enabling robots to follow abstract instructions and complete complex dynamic tasks

AI Meets Physical World -- Exploring Robot Cooking

Decision-Making in Robotic Grasping with Large Language Models.

Realtime Motion Generation with Active Perception Using Attention Mechanism for Cooking Robot

Audio-visual scene understanding utilizing text information for a cooking support robot

RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools

Combining Ontological Knowledge and Large Language Model for User-Friendly Service Robots

Learning Sequential Acquisition Policies for Robot-Assisted Feeding

LLM-based Robot Task Planning with Exceptional Handling for General Purpose Service Robots

Task Planning for Robot Manipulator Using Natural Language Task Input with Large Language Models

Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos

Open-vocabulary Queryable Scene Representations for Real World Planning

PizzaCommonSense: Learning to Model Commonsense Reasoning about Intermediate Steps in Cooking Recipes

VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks