Ego-perspective enhanced fitness training experience of AR Try to Move game

Chongyu Zhang

2023-09-26

Abstract:AR, a recent emerging technology, has been widely used in entertainment to provide users with immersive, interactive, and, sometimes, engaging experiences. The process of rehabilitation treatment and motor training process is often boring, and it is well known that users' exercise efficiency is often not as efficient as in a rehabilitation institution. Thus far, there is no effective upper limb sports rehabilitation training game based on the ego-perspective. Hence, with the objective of enhancing the enjoyment experience in rehabilitation and more effective remote rehabilitation training, this work aims to provide an AR Try to Move game and a convolutional neural network (CNN) for identifying and classifying user gestures from a self-collected AR multiple interactive gestures dataset. Utilizing an AR game scoring system, users are incentivized to enhance their upper limb muscle system through remote training with greater effectiveness and convenience.

Human-Computer Interaction,Artificial Intelligence,Image and Video Processing

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are: **How to improve the interestingness and effectiveness of rehabilitation training through augmented reality (AR) technology, especially for upper - limb motor rehabilitation training**. Specifically, the existing rehabilitation treatment and sports training processes are usually rather dull, resulting in users' exercise efficiency not being as high as that in professional rehabilitation institutions. In addition, there are currently no effective upper - limb motor rehabilitation training games based on the ego - perspective.

To solve these problems, the author proposes a game named **AR Try to Move** and combines convolutional neural networks (CNN) to recognize and classify users' hand gestures. This game aims to enhance the effectiveness of rehabilitation training in the following ways:

1. **Enhancing interestingness**: By designing an AR jigsaw puzzle game that includes multiple interactive hand gestures, users can perform diverse upper - limb movements during the task - completion process, increasing the interestingness and participation in training.
2. **Remote rehabilitation training**: Using AR technology and a scoring system, users can conduct efficient remote rehabilitation training at home without having to go to professional institutions.
3. **Real - time feedback and motivation**: By classifying hand gestures through CNN and providing immediate feedback, users are encouraged to perform more interactive hand gestures and muscle activations, thereby improving the training effect.

### Specific implementation methods:
- **AR Try to Move game design**: The game contains four different difficulty levels. Users need to move the colored squares into the blue target containers, and each interaction action will trigger different upper - limb movements.
- **Hand - gesture data set collection and classification**: Collected multiple interactive hand - gesture data of users in the game and used a lightweight CNN model to classify these hand gestures into 16 categories.
- **Scoring system**: Calculate the basic score, hand - gesture - times bonus, and muscle - activation bonus to finally obtain the total score, so as to motivate users to complete the game more efficiently.

### Experimental results:
Experiments show that this game mechanism can effectively motivate users to perform more interactive hand gestures and spatial movements, shorten the completion time, and obtain a higher total score. This not only improves the interestingness of training but also helps users better understand their muscle - activation states, achieving a training effect similar to that of traditional rehabilitation institutions.

### Future work:
Future research directions include configuring specific stimulation signals for patients who are unable to move independently to help them complete the game and further expanding the application range of the system to meet the rehabilitation needs under different medical conditions.

Through these methods, this research provides a new solution for remote rehabilitation training in digital medicine and is expected to improve patients' rehabilitation experiences and effects.