Characterization of Human Balance through a Reinforcement Learning-based Muscle Controller

Kübra Akbaş,Carlotta Mummolo,Xianlian Zhou

2023-08-08

Abstract:Balance assessment during physical rehabilitation often relies on rubric-oriented battery tests to score a patient's physical capabilities, leading to subjectivity. While some objective balance assessments exist, they are often limited to tracking the center of pressure (COP), which does not fully capture the whole-body postural stability. This study explores the use of the center of mass (COM) state space and presents a promising avenue for monitoring the balance capabilities in humans. We employ a musculoskeletal model integrated with a balance controller, trained through reinforcement learning (RL), to investigate balancing capabilities. The RL framework consists of two interconnected neural networks governing balance recovery and muscle coordination respectively, trained using Proximal Policy Optimization (PPO) with reference state initialization, early termination, and multiple training strategies. By exploring recovery from random initial COM states (position and velocity) space for a trained controller, we obtain the final BR enclosing successful balance recovery trajectories. Comparing the BRs with analytical postural stability limits from a linear inverted pendulum model, we observe a similar trend in successful COM states but more limited ranges in the recoverable areas. We further investigate the effect of muscle weakness and neural excitation delay on the BRs, revealing reduced balancing capability in different regions. Overall, our approach of learning muscular balance controllers presents a promising new method for establishing balance recovery limits and objectively assessing balance capability in bipedal systems, particularly in humans.

Machine Learning,Robotics,Systems and Control

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the limitations of current human body balance assessment methods. Specifically:

1. **Subjectivity problem**: In the current physical rehabilitation process, balance assessment usually depends on scoring criteria, which leads to the subjectivity of assessment results. For example, some commonly used balance tests (such as Berg Balance Scale, Mini BESTest, etc.) have a certain degree of objectivity, but are still affected by the subjective judgment of raters.

2. **Limitations of assessment indicators**: Existing objective balance assessment methods mainly focus on tracking the center - of - pressure (COP), but this cannot comprehensively reflect the whole - body postural stability. COP can only provide limited information and cannot capture the overall body dynamics and muscle activities, so it cannot fully predict the risk of falling.

3. **Lack of understanding of muscle control mechanisms**: Traditional assessment methods are mainly based on kinematic and kinetic information, while ignoring the important influence of muscle control mechanisms on human body balance. Especially for patients with neuromuscular diseases, changes in muscle parameters may affect their balance reactions, and these changes are difficult to be captured by traditional methods.

To solve these problems, this research proposes a new method, that is, to assess and monitor human body balance ability through a muscle controller trained by reinforcement learning (RL). This method utilizes the human musculoskeletal model and optimizes the controller through RL algorithms, enabling it to restore balance in different initial states. Specifically, this research aims to:

- **Establish the balance region (BR)**: By exploring successful balance restoration trajectories in different initial states, construct a region that describes the limits of human body balance.

- **Evaluate the effects of muscle weakening and nerve delay**: Study how these factors affect balance ability, so as to provide a deeper understanding for clinical diagnosis and rehabilitation treatment.

- **Provide a more objective balance assessment method**: By combining COM state - space analysis and RL - trained muscle controllers, provide a more objective and personalized balance assessment means, especially in bipedal systems (such as humans).

In short, the goal of this research is to improve the deficiencies of existing balance assessment methods by introducing advanced computational models and technologies, and to provide better tools and support for the medical and rehabilitation fields.