Abstract:We are developing a system for long term Semi-Automated Rehabilitation At the Home (SARAH) that relies on low-cost and unobtrusive video-based sensing. We present a cyber-human methodology used by the SARAH system for automated assessment of upper extremity stroke rehabilitation at the home. We propose a hierarchical model for automatically segmenting stroke survivor's movements and generating training task performance assessment scores during rehabilitation. The hierarchical model fuses expert therapist knowledge-based approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high level constraints relating to activity structure (i.e., type and order of segments per task). We utilize an HMM and a Decision Tree model to connect these high level priors to data driven analysis. The lower layers (RGB images and raw kinematics) need to be addressed primarily through data driven techniques. We use a transformer based architecture operating on low-level action features (tracking of individual body joints and objects) and a Multi-Stage Temporal Convolutional Network(MS-TCN) operating on raw RGB images. We develop a sequence combining these complimentary algorithms effectively, thus encoding the information from different layers of the movement hierarchy. Through this combination, we produce a robust segmentation and task assessment results on noisy, variable and limited data, which is characteristic of low cost video capture of rehabilitation at the home. Our proposed approach achieves 85% accuracy in per-frame labeling, 99% accuracy in segment classification and 93% accuracy in task completion assessment. Although the methodology proposed in this paper applies to upper extremity rehabilitation using the SARAH system, it can potentially be used, with minor alterations, to assist automation in many other movement rehabilitation contexts (i.e., lower extremity training for neurological accidents).

Real-time AI-assisted visual exercise pose correctness during rehabilitation training for musculoskeletal disorder

Realtime Indoor Workout Analysis Using Machine Learning & Computer Vision

AI Coach: Deep Human Pose Estimation and Analysis for Personalized Athletic Training Assistance

3D Pose Based Feedback for Physical Exercises

Full-body pose reconstruction and correction in virtual reality for rehabilitation training

An Upper Limb Rehabilitation Exercise Status Identification System Based on Machine Learning and IoT

A real-time algorithm for weight training detection and correction

Multiview Human Pose Estimation and Corrective Feedback

Rehabilitation Exercise Repetition Segmentation and Counting using Skeletal Body Joints

Real-Time Posture Monitoring and Risk Assessment for Manual Lifting Tasks Using MediaPipe and LSTM

A Skeleton-Based Rehabilitation Exercise Assessment System with Rotation Invariance

Real-Time Human Pose Estimation on a Smart Walker using Convolutional Neural Networks

UCO Physical Rehabilitation: New Dataset and Study of Human Pose Estimation Methods on Physical Rehabilitation Exercises

Physiotherapy Exercise Classification with Single-Camera Pose Detection and Machine Learning

Improved spatial-temporal graph convolutional networks for upper limb rehabilitation assessment based on precise posture measurement

Automated Movement Assessment in Stroke Rehabilitation

Automatic Classification of Knee Rehabilitation Exercises Using a Single Inertial Sensor: a Case Study

Fitness Done Right: a Real-time Intelligent Personal Trainer for Exercise Correction

Aasna: Kinematic Yoga Posture Detection And Correction System Using CNN

Real-time Recognition of Yoga Poses using computer Vision for Smart Health Care

Video based exercise recognition and correct pose detection