Abstract:Abstract The redundant kinematic structure of the human upper body provides sufficient dexterity in performing daily living activities, such as drinking, eating, and other manipulation tasks. Most of these Activities of Daily Living (ADL) tasks can be performed using various trajectories; however, healthy humans employ some possible trajectories to accomplish these tasks. Assistive robots can explore these Cartesian trajectories and joint angles’ trajectories from human demonstrations and utilize artificial intelligence techniques to recognize, learn, and perform ADLs using robotic arms. This paper aims to develop a platform for Assistive Robots to identify and perform ADLs through human demonstration. In this paper, two Mask Regional Convolutional Neural Networks (Mask R-CNN) were developed to predict human intention for a given task, and the Reinforcement Learning approach was implemented for optimum trajectory planning. The First Mask R-CNN network, which was used for the detection of an object, was trained on a novel dataset. This dataset comprises images captured by the research team as well as gathered images from the internet. The first Mask R-CNN is expected to achieve two main objectives. The first objective is related to structure learning, which detects an object in a cluttered environment. The second objective is to estimate parameters that maximize the weight and detection rate of diverse types of similar objects. The first Mask R-CNN tries to detect the object with its location in the image and provides a soft mask on it. The second Mask R-CNN, which tries to detect human posture, was trained on a pre-existing dataset comprising human posture images. The output of these two Mask R-CNN’s then feed-forward to recognize human intensions that can be achieved through human-pose key points. Prediction of human intention was performed if the detected object and human pose match a range of pre-stored values specific to ADLs. Once the human intention has been identified via Mask R-CNN networks, the system starts planning for an optimal trajectory. For the trajectory planning, the Q-learning approach was implemented, which uses the concept of reward and penalty while exploring the unstructured environment in order to find an optimal trajectory. Experiments were conducted to determine the level of accuracy and reliability provided by our assistive platform in predicting human actions and performing trajectory planning. Results show that our assistive platform successfully recognized human actions such as water pouring, cereal making, and opening the door, as well as finding optimized trajectories to perform those actions.

Prediction of Intentions Behind a Single Human Action: an Application of Convolutional Neural Network

Better Than Humans: a Method for Inferring Consumer Shopping Intentions by Reading Facial Expressions

A brain-inspired intention prediction model and its applications to humanoid robot

Human Action Recognition From Digital Videos Based on Deep Learning.

An Attention-Based Deep Learning Approach for Inertial Motion Recognition and Estimation in Human-Robot Collaboration

Inferring Human Intentions from Predicted Action Probabilities

Hybrid Recurrent Neural Network Architecture-Based Intention Recognition for Human-Robot Collaboration

A Spatio-Temporal Transformer Network for Human Motion Prediction in Human-Robot Collaboration

Human Action Recognition Using Deep Learning Methods.

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Intention Inference for Space Targets Using Deep Convolutional Neural Network

Robot Learning From Human Demonstration of Activities of Daily Living (ADL) Tasks

Optimized, robust, real-time emotion prediction for human-robot interactions using deep learning

Comparison of LSTM, Transformers, and MLP-mixer neural networks for gaze based human intention prediction

A Vision-Based Measure of Environmental Effects on Inferring Human Intention During Human Robot Interaction

Spiking Neural Networks for Early Prediction in Human Robot Collaboration

Early Turn-taking Prediction with Spiking Neural Networks for Human Robot Collaboration

Enhancing Robotic Collaborative Tasks Through Contextual Human Motion Prediction and Intention Inference

Hybrid machine learning for human action recognition and prediction in assembly

Convolutional neural network-based intention forecasting and lane change path predicting of the human driver

Cross-domain prediction approach of human lower limb voluntary movement intention for exoskeleton robot based on EEG signals