Terrain-Aware Stride-Level Trajectory Forecasting for a Powered Hip Exoskeleton via Vision and Kinematics Fusion

Ruoqi Zhao,Xingbang Yan,Yubo Fan

2024-04-18

Abstract:Powered hip exoskeletons have shown the ability for locomotion assistance during treadmill walking. However, providing suitable assistance in real-world walking scenarios which involve changing terrain remains challenging. Recent research suggests that forecasting the lower limb joint's angles could provide target trajectories for exoskeletons and prostheses, and the performance could be improved with visual information. In this letter, We share a real-world dataset of 10 healthy subjects walking through five common types of terrain with stride-level label. We design a network called Sandwich Fusion Transformer for Image and Kinematics (SFTIK), which predicts the thigh angle of the ensuing stride given the terrain images at the beginning of the preceding and the ensuing stride and the IMU time series during the preceding stride. We introduce width-level patchify, tailored for egocentric terrain images, to reduce the computational demands. We demonstrate the proposed sandwich input and fusion mechanism could significantly improve the forecasting performance. Overall, the SFTIK outperforms baseline methods, achieving a computational efficiency of 3.31 G Flops, and root mean square error (RMSE) of 3.445 \textpm \ 0.804\textdegree \ and Pearson's correlation coefficient (PCC) of 0.971 \textpm\ 0.025. The results demonstrate that SFTIK could forecast the thigh's angle accurately with low computational cost, which could serve as a terrain adaptive trajectory planning method for hip exoskeletons. Codes and data are available at

Robotics

What problem does this paper attempt to address?

This paper aims to address the problem of providing adaptive motion assistance for powered hip exoskeletons in different terrains. Existing exoskeleton control strategies perform poorly in real walking scenarios because they fail to effectively adapt to continuously changing terrains. The researchers propose a network called "Sandwich Fusion Transformer for Image and Kinematics" (SFTIK), which combines visual images and inertial measurement unit (IMU) data to predict the thigh angle for the next gait cycle. By using a method called width-level partitioning, the computational requirements for first-person terrain images are reduced, resulting in improved prediction performance. The paper presents a dataset consisting of walking data for 10 healthy participants in 5 common terrains, with labels for each gait cycle. The SFTIK network utilizes terrain images of the previous and next steps, as well as the IMU time series of the previous step, to predict the thigh angle. Its design includes a "sandwich" fusion mechanism that significantly improves prediction performance. Experimental results show that SFTIK achieves high prediction accuracy (RMSE of 3.445±0.804°, PCC of 0.971±0.025) while maintaining low computational costs, making it a promising approach for adaptive trajectory planning of hip exoskeletons in different terrains.

Terrain-Aware Stride-Level Trajectory Forecasting for a Powered Hip Exoskeleton via Vision and Kinematics Fusion

Fuzzy-Logic-Based Terrain Identification with Multisensor Fusion for Transtibial Amputees

Adaptive Gait Trajectory and Event Prediction of Lower Limb Exoskeletons for Various Terrains Using Reinforcement Learning

Enhancing Prosthetic Safety and Environmental Adaptability: A Visual-Inertial Prosthesis Motion Estimation Approach on Uneven Terrains

Feature Decoupling for Multimodal Locomotion and Estimation of Knee and Ankle Angles Implemented by Multi-Model Fusion

A fuzzy logic based terrain identification approach to prosthesis control using multi-sensor fusion

The Visual Footsteps Planning System for Exoskeleton Robots under Complex Terrain

A Wearable Computer Vision System With Gimbal Enables Position-, Speed-, and Phase-Independent Terrain Classification for Lower Limb Prostheses

Motion Forecasting Network (mofcnet): IMU-Based Human Motion Forecasting for Hip Assistive Exoskeleton

Real Time Gait Planning for a Mobile Medical Exoskeleton with Crutche.

CROSS-GAiT: Cross-Attention-Based Multimodal Representation Fusion for Parametric Gait Adaptation in Complex Terrains

Real-Time Gait Phase and Task Estimation for Controlling a Powered Ankle Exoskeleton on Extremely Uneven Terrain

A Terrain-based Gait Self-adjusting Planning for Powered Prostheses

A Transformer-Based Neural Network for Gait Prediction in Lower Limb Exoskeleton Robots Using Plantar Force

Impact-Aware Foot Motion Reconstruction and Ramp/Stair Detection Using One Foot-Mounted Inertial Measurement Unit

Beyond Gait: Learning Knee Angle for Seamless Prosthesis Control in Multiple Scenarios

Terrain slope parameter recognition for exoskeleton robot in urban multi-terrain environments

Personalization of Wearable Sensor-Based Joint Kinematic Estimation Using Computer Vision for Hip Exoskeleton Applications

Terrain Identification for Prosthetic Knees Based on Electromyographic Signal Features

iP3T: an interpretable multimodal time-series model for enhanced gait phase prediction in wearable exoskeletons

Improving IMU-Based Prediction of Lower Limb Kinematics in Natural Environments Using Egocentric Optical Flow