Abstract:Wearable egocentric cameras and machine learning have the potential to provide clinicians with a more nuanced understanding of patient hand use at home after stroke and spinal cord injury (SCI). However, they require detailed contextual information (i.e., activities and object interactions) to effectively interpret metrics and meaningfully guide therapy planning. We demonstrate that an object-centric approach, focusing on what objects patients interact with rather than how they move, can effectively recognize Activities of Daily Living (ADL) in real-world rehabilitation settings. We evaluated our models on a complex dataset collected in the wild comprising 2261 minutes of egocentric video from 16 participants with impaired hand function. By leveraging pre-trained object detection and hand-object interaction models, our system achieves robust performance across different impairment levels and environments, with our best model achieving a mean weighted F1-score of 0.78 +/- 0.12 and maintaining an F1-score > 0.5 for all participants using leave-one-subject-out cross validation. Through qualitative analysis, we observe that this approach generates clinically interpretable information about functional object use while being robust to patient-specific movement variations, making it particularly suitable for rehabilitation contexts with prevalent upper limb impairment.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in the outpatient neuro - rehabilitation environment, how to accurately identify patients' activities of daily living (ADL) at home through wearable first - person cameras and machine - learning techniques, in order to provide more detailed information on the use of hand functions, thereby providing meaningful treatment - planning information for clinicians. Specifically, the author points out that current clinical assessment methods rely on direct observation and patients' self - reports. These methods have recall biases and can only provide snapshots of functions in the clinical environment, unable to capture the diversity and compensation strategies of patients at home. This has led to a gap between clinical assessment and actual home functions, posing a major challenge to the design of targeted rehabilitation interventions. To solve these problems, the author proposes an object - centric method, focusing on which objects patients interact with rather than how they move. This method aims to: 1. **Flexibility**: Identify activity categories based on object - interaction patterns without the need for predefined specific activities. 2. **Interpretability**: Provide clear information on the use of functional objects, making the results consistent with clinical assessment methods. 3. **Feasibility**: It can be deployed using pre - trained object - detection models without the need for patient - specific training data. Through this method, the author hopes to supplement the existing hand - function analysis framework and provide clinicians with detailed background information on the frequency and quality of hand use, thereby better guiding rehabilitation treatment.

Detecting Activities of Daily Living in Egocentric Video to Contextualize Hand Use at Home in Outpatient Neurorehabilitation Settings