StairNet: Visual Recognition of Stairs for Human-Robot Locomotion

Andrew Garrett Kurbis,Dmytro Kuzmenko,Bogdan Ivanyuk-Skulskiy,Alex Mihailidis,Brokoslaw Laschowski

2023-11-01

Abstract:Human-robot walking with prosthetic legs and exoskeletons, especially over complex terrains such as stairs, remains a significant challenge. Egocentric vision has the unique potential to detect the walking environment prior to physical interactions, which can improve transitions to and from stairs. This motivated us to create the StairNet initiative to support the development of new deep learning models for visual sensing and recognition of stairs, with an emphasis on lightweight and efficient neural networks for onboard real-time inference. In this study, we present an overview of the development of our large-scale dataset with over 515,000 manually labeled images, as well as our development of different deep learning models (e.g., 2D and 3D CNN, hybrid CNN and LSTM, and ViT networks) and training methods (e.g., supervised learning with temporal data and semi-supervised learning with unlabeled images) using our new dataset. We consistently achieved high classification accuracy (i.e., up to 98.8%) with different designs, offering trade-offs between model accuracy and size. When deployed on mobile devices with GPU and NPU accelerators, our deep learning models achieved inference speeds up to 2.8 ms. We also deployed our models on custom-designed CPU-powered smart glasses. However, limitations in the embedded hardware yielded slower inference speeds of 1.5 seconds, presenting a trade-off between human-centered design and performance. Overall, we showed that StairNet can be an effective platform to develop and study new visual perception systems for human-robot locomotion with applications in exoskeleton and prosthetic leg control.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of visual recognition when robots assist humans in walking on complex terrains (such as stairs). Specifically, the research team developed an initiative called StairNet, with the goal of supporting the development of new deep learning models, particularly lightweight and efficient neural networks capable of real-time stair environment recognition. The study not only created a large-scale dataset containing over 515,000 manually annotated images but also developed various deep learning models (such as 2D and 3D convolutional neural networks, hybrid CNN and LSTM models, and vision transformer networks). Different training methods, including supervised learning and semi-supervised learning, were explored using the new dataset.

Through these efforts, the researchers hope to improve the navigation capabilities of robot-assisted devices (such as prosthetic legs and exoskeletons) in complex environments like stairs, thereby reducing the risk of falls due to environmental misjudgment and enhancing the human-machine interaction experience. Additionally, they paid special attention to the real-time inference performance of the models on mobile devices to ensure that the system can make quick and accurate decisions in practical applications. In summary, the goal of this research is to establish an effective platform for developing and studying new visual perception systems to support the advancement of robot-assisted walking technology.