Abstract:Abstract The redundant kinematic structure of the human upper body provides sufficient dexterity in performing daily living activities, such as drinking, eating, and other manipulation tasks. Most of these Activities of Daily Living (ADL) tasks can be performed using various trajectories; however, healthy humans employ some possible trajectories to accomplish these tasks. Assistive robots can explore these Cartesian trajectories and joint angles’ trajectories from human demonstrations and utilize artificial intelligence techniques to recognize, learn, and perform ADLs using robotic arms. This paper aims to develop a platform for Assistive Robots to identify and perform ADLs through human demonstration. In this paper, two Mask Regional Convolutional Neural Networks (Mask R-CNN) were developed to predict human intention for a given task, and the Reinforcement Learning approach was implemented for optimum trajectory planning. The First Mask R-CNN network, which was used for the detection of an object, was trained on a novel dataset. This dataset comprises images captured by the research team as well as gathered images from the internet. The first Mask R-CNN is expected to achieve two main objectives. The first objective is related to structure learning, which detects an object in a cluttered environment. The second objective is to estimate parameters that maximize the weight and detection rate of diverse types of similar objects. The first Mask R-CNN tries to detect the object with its location in the image and provides a soft mask on it. The second Mask R-CNN, which tries to detect human posture, was trained on a pre-existing dataset comprising human posture images. The output of these two Mask R-CNN’s then feed-forward to recognize human intensions that can be achieved through human-pose key points. Prediction of human intention was performed if the detected object and human pose match a range of pre-stored values specific to ADLs. Once the human intention has been identified via Mask R-CNN networks, the system starts planning for an optimal trajectory. For the trajectory planning, the Q-learning approach was implemented, which uses the concept of reward and penalty while exploring the unstructured environment in order to find an optimal trajectory. Experiments were conducted to determine the level of accuracy and reliability provided by our assistive platform in predicting human actions and performing trajectory planning. Results show that our assistive platform successfully recognized human actions such as water pouring, cereal making, and opening the door, as well as finding optimized trajectories to perform those actions.

What Would You Do? Acting by Learning to Predict

Robotic Imitation of Human Actions

Modeling Long-horizon Tasks as Sequential Interaction Landscapes

Watch and Act: Learning Robotic Manipulation from Visual Demonstration.

Learning to Imagine Manipulation Goals for Robot Task Planning

Combined Task and Action Learning from Human Demonstrations for Mobile Manipulation Applications

Concept2Robot: Learning Manipulation Concepts from Instructions and Human Demonstrations

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

What Can I Do Here? Learning New Skills by Imagining Visual Affordances

Efficient multitask learning with an embodied predictive model for door opening and entry with whole-body control

Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

Ergodic imitation: Learning from what to do and what not to do

Learning One-Shot Imitation From Humans Without Humans

Learning Multi-Step Manipulation Tasks from A Single Human Demonstration

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

Robot Learning From Human Demonstration of Activities of Daily Living (ADL) Tasks

Learning to combine primitive skills: A step towards versatile robotic manipulation

Learning Skills from Demonstrations: A Trend from Motion Primitives to Experience Abstraction

Learning Robotic Manipulation through Visual Planning and Acting

Vision-based Robot Manipulation Learning via Human Demonstrations

Visual Robot Task Planning