Abstract:Most of the humanoid social robots currently diffused are designed only for verbal and animated interactions with users, and despite being equipped with two upper arms for interactive animation, they lack object manipulation capabilities. In this paper, we propose the MONOCULAR (eMbeddable autONomous ObjeCt manipULAtion Routines) framework, which implements a set of routines to add manipulation functionalities to social robots by exploiting the functional data fusion of two RGB cameras and a 3D depth sensor placed in the head frame. The framework is designed to: (i) localize specific objects to be manipulated via RGB cameras; (ii) define the characteristics of the shelf on which they are placed; and (iii) autonomously adapt approach and manipulation routines to avoid collisions and maximize grabbing accuracy. To localize the item on the shelf, MONOCULAR exploits an embeddable version of the You Only Look Once (YOLO) object detector. The RGB camera outcomes are also used to estimate the height of the shelf using an edge-detecting algorithm. Based on the item's position and the estimated shelf height, MONOCULAR is designed to select between two possible routines that dynamically optimize the approach and object manipulation parameters according to the real-time analysis of RGB and 3D sensor frames. These two routines are optimized for a central or lateral approach to objects on a shelf. The MONOCULAR procedures are designed to be fully automatic, intrinsically protecting sensitive users' data and stored home or hospital maps. MONOCULAR was optimized for Pepper by SoftBank Robotics. To characterize the proposed system, a case study in which Pepper is used as a drug delivery operator is proposed. The case study is divided into: (i) pharmaceutical package search; (ii) object approach and manipulation; and (iii) delivery operations. Experimental data showed that object manipulation routines for laterally placed objects achieves a best grabbing success rate of 96%, while the routine for centrally placed objects can reach 97% for a wide range of different shelf heights. Finally, a proof of concept is proposed here to demonstrate the applicability of the MONOCULAR framework in a real-life scenario.

EfficientPPS: Part-aware Panoptic Segmentation of Transparent Objects for Robotic Manipulation

EfficientLPS: Efficient LiDAR Panoptic Segmentation

APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation

Object-Independent Human-to-Robot Handovers Using Real Time Robotic Vision

RISeg: Robot Interactive Object Segmentation via Body Frame-Invariant Features

RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter

SG-Grasp: Semantic Segmentation Guided Robotic Grasp Oriented to Weakly Textured Objects Based on Visual Perception Sensors

Bin-picking of novel objects through category-agnostic-segmentation: RGB matters

LucidGrasp: Robotic Framework for Autonomous Manipulation of Laboratory Equipment with Different Degrees of Transparency via 6D Pose Estimation

The Sum of Its Parts: Visual Part Segmentation for Inertial Parameter Identification of Manipulated Objects

OptiGrasp: Optimized Grasp Pose Detection Using RGB Images for Warehouse Picking Robots

Real-time 3D Semantic Scene Perception for Egocentric Robots with Binocular Vision

Cascade contour-enhanced panoptic segmentation for robotic vision perception

Point Cloud Hand–Object Segmentation Using Multimodal Imaging with Thermal and Color Data for Safe Robotic Object Handover

A Preliminary Exploration to Make Stereotactic Surgery Robots Aware of the Semantic 2D/3D Working Scene

[Recognizing transparent objects for laboratory automation]

Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations

Vision-based Manipulation of Transparent Plastic Bags in Industrial Setups

An Embedded Framework for Fully Autonomous Object Manipulation in Robotic-Empowered Assisted Living

Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgery