Abstract:Robot-assisted feeding has the potential to improve the quality of life for individuals with mobility limitations who are unable to feed themselves independently. However, there exists a large gap between the homogeneous, curated plates existing feeding systems can handle, and truly in-the-wild meals. Feeding realistic plates is immensely challenging due to the sheer range of food items that a robot may encounter, each requiring specialized manipulation strategies which must be sequenced over a long horizon to feed an entire meal. An assistive feeding system should not only be able to sequence different strategies efficiently in order to feed an entire meal, but also be mindful of user preferences given the personalized nature of the task. We address this with FLAIR, a system for long-horizon feeding which leverages the commonsense and few-shot reasoning capabilities of foundation models, along with a library of parameterized skills, to plan and execute user-preferred and efficient bite sequences. In real-world evaluations across 6 realistic plates, we find that FLAIR can effectively tap into a varied library of skills for efficient food pickup, while adhering to the diverse preferences of 42 participants without mobility limitations as evaluated in a user study. We demonstrate the seamless integration of FLAIR with existing bite transfer methods [19, 28], and deploy it across 2 institutions and 3 robots, illustrating its adaptability. Finally, we illustrate the real-world efficacy of our system by successfully feeding a care recipient with severe mobility limitations. Supplementary materials and videos can be found at: <a class="link-external link-https" href="https://emprise.cs.cornell.edu/flair" rel="external noopener nofollow">this https URL</a> .

DishAgent: Enhancing Dining Experiences Through LLM-Based Smart Dishes

"Mango Mango, How to Let The Lettuce Dry Without A Spinner?'': Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking Partner

AI-Gadget Kit: Integrating Swarm User Interfaces with LLM-driven Agents for Rich Tabletop Game Applications

Deep-Learning-Assisted Multi-Dish Food Recognition Application for Dietary Intake Reporting

PIC2DISH: A Customized Cooking Assistant System.

Towards an LLM-Based Speech Interface for Robot-Assisted Feeding

Mixed Dining: Enhancing Social Interaction of Elderly Individuals Living Alone with Smart Dining Environment and Wearable MR

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

Roll Up Your Sleeves: Working with a Collaborative and Engaging Task-Oriented Dialogue System

LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application

FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishes

MindAgent: Emergent Gaming Interaction

Kitchen Artist: Precise Control of Liquid Dispensing for Gourmet Plating

MOSAIC: A Modular System for Assistive and Interactive Cooking

Simulating User Agents for Embodied Conversational-AI

The Deep Features and Attention Mechanism-Based Method to Dish Healthcare Under Social IoT Systems: An Empirical Study With a Hand-Deep Local–Global Net

Human-Centered LLM-Agent User Interface: A Position Paper

ChatDiet: Empowering Personalized Nutrition-Oriented Food Recommender Chatbots through an LLM-Augmented Framework

DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Control

Turn Every Application into an Agent: Towards Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

The Design and Evaluation of User-Interface Prototypes for a Next-Generation Dishwasher Mobile Application