Abstract:Robotic assistive feeding holds significant promise for improving the quality of life for individuals with eating disabilities. However, acquiring diverse food items under varying conditions and generalizing to unseen food presents unique challenges. Existing methods that rely on surface-level geometric information (e.g., bounding box and pose) derived from visual cues (e.g., color, shape, and texture) often lacks adaptability and robustness, especially when foods share similar physical properties but differ in visual appearance. We employ imitation learning (IL) to learn a policy for food acquisition. Existing methods employ IL or Reinforcement Learning (RL) to learn a policy based on off-the-shelf image encoders such as ResNet-50. However, such representations are not robust and struggle to generalize across diverse acquisition scenarios. To address these limitations, we propose a novel approach, IMRL (Integrated Multi-Dimensional Representation Learning), which integrates visual, physical, temporal, and geometric representations to enhance the robustness and generalizability of IL for food acquisition. Our approach captures food types and physical properties (e.g., solid, semi-solid, granular, liquid, and mixture), models temporal dynamics of acquisition actions, and introduces geometric information to determine optimal scooping points and assess bowl fullness. IMRL enables IL to adaptively adjust scooping strategies based on context, improving the robot's capability to handle diverse food acquisition scenarios. Experiments on a real robot demonstrate our approach's robustness and adaptability across various foods and bowl configurations, including zero-shot generalization to unseen settings. Our approach achieves improvement up to $35\%$ in success rate compared with the best-performing baseline.

Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding

Robot-Assisted Feeding: Generalizing Skewering Strategies across Food Items on a Realistic Plate

Playing with Food: Learning Food Item Representations through Interactive Exploration

Towards Robotic Feeding: Role of Haptics in Fork-Based Food Manipulation

Adaptive Robot-Assisted Feeding: An Online Learning Framework for Acquiring Previously Unseen Food Items

Learning Sequential Acquisition Policies for Robot-Assisted Feeding

Learning Bimanual Scooping Policies for Food Acquisition

FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishes

IMRL: Integrating Visual, Physical, Temporal, and Geometric Representations for Enhanced Food Acquisition

Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding

Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Types

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

Vision-based food handling system for high-resemblance random food items

Cluttered Food Grasping with Adaptive Fingers and Synthetic-Data Trained Object Detection

Leveraging Multimodal Haptic Sensory Data for Robust Cutting

Balancing Efficiency and Comfort in Robot-Assisted Bite Transfer

Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulation

Kiri-Spoon: A Soft Shape-Changing Utensil for Robot-Assisted Feeding

REPeat: A Real2Sim2Real Approach for Pre-acquisition of Soft Food Items in Robot-assisted Feeding

Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing

Robotic Handling of Compliant Food Objects by Robust Learning from Demonstration