REST-HANDS: Rehabilitation with Egocentric Vision Using Smartglasses for Treatment of Hands after Surviving Stroke

Wiktor Mucha,Kentaro Tanaka,Martin Kampel

2024-09-30

Abstract:Stroke represents the third cause of death and disability worldwide, and is recognised as a significant global health problem. A major challenge for stroke survivors is persistent hand dysfunction, which severely affects the ability to perform daily activities and the overall quality of life. In order to regain their functional hand ability, stroke survivors need rehabilitation therapy. However, traditional rehabilitation requires continuous medical support, creating dependency on an overburdened healthcare system. In this paper, we explore the use of egocentric recordings from commercially available smart glasses, specifically RayBan Stories, for remote hand rehabilitation. Our approach includes offline experiments to evaluate the potential of smart glasses for automatic exercise recognition, exercise form evaluation and repetition counting. We present REST-HANDS, the first dataset of egocentric hand exercise videos. Using state-of-the-art methods, we establish benchmarks with high accuracy rates for exercise recognition (98.55%), form evaluation (86.98%), and repetition counting (mean absolute error of 1.33). Our study demonstrates the feasibility of using egocentric video from smart glasses for remote rehabilitation, paving the way for further research.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the remote rehabilitation treatment of hand dysfunction in stroke survivors. Specifically, the authors explored the use of commercial smart glasses (such as RayBan Stories) to capture first - person - view videos for automated hand rehabilitation assessment. Key challenges in the paper include:

1. **Identification of Hand Rehabilitation Exercises**: Through the first - person - view videos captured by smart glasses, the system needs to be able to automatically identify the type of hand rehabilitation exercise that the patient is performing.
2. **Assessment of Exercise Form**: The system needs to be able to assess whether the form in which the patient performs the hand rehabilitation exercise is correct to ensure the rehabilitation effect.
3. **Counting of Repetition Times**: The system also needs to be able to automatically count the number of repetitions of each exercise in order to track the patient's rehabilitation progress.

### Main Contributions
- **Proposing the Use of First - Person - View Videos for Remote Hand Rehabilitation Assessment**: The authors proposed using first - person - view videos captured by comfortable smart glasses (such as RayBan Stories) for automated remote hand rehabilitation assessment.
- **Creating the First Hand Rehabilitation Exercise Dataset**: The authors created the REST - HANDS dataset, which is the first hand rehabilitation exercise dataset recorded from the first - person - view, containing videos and their labels, including exercise categories, forms evaluated by professional physiotherapists, and the number of repetitions of each exercise.
- **Applying State - of - the - Art Video Understanding Methods**: The authors used the latest video understanding methods, including 3D convolutional neural networks (3D CNN) and Transformer - based architectures, to achieve the identification of hand rehabilitation exercises, form assessment, and repetition - times counting.
- **Extensive Experiments and Evaluations**: The authors conducted extensive experiments and evaluations on the proposed system, demonstrating its feasibility in hand rehabilitation exercise identification (with an accuracy of 98.55%), form assessment (with an accuracy of 86.98%), and repetition - times counting (with a mean absolute error of 1.33), and identified key areas for future research.

### Conclusion
This research demonstrates the feasibility of using first - person - view videos for remote hand rehabilitation assessment, laying the foundation for future related research and technological development. In an automated manner, such a system can reduce the participation of medical personnel, relieve the burden on the medical system, and at the same time improve the rehabilitation effect and quality of life of stroke survivors.