Abstract:Human–robot walking with prosthetic legs and exoskeletons, especially over complex terrains, such as stairs, remains a significant challenge. Egocentric vision has the unique potential to detect the walking environment prior to physical interactions, which can improve transitions to and from stairs. This motivated us to develop the StairNet initiative to support the development of new deep learning models for visual perception of real-world stair environments. In this study, we present a comprehensive overview of the StairNet initiative and key research to date. First, we summarize the development of our large-scale data set with over 515,000 manually labeled images. We then provide a summary and detailed comparison of the performances achieved with different algorithms (i.e., 2D and 3D CNN, hybrid CNN and LSTM, and ViT networks), training methods (i.e., supervised learning with and without temporal data, and semi-supervised learning with unlabeled images), and deployment methods (i.e., mobile and embedded computing), using the StairNet data set. Finally, we discuss the challenges and future directions. To date, our StairNet models have consistently achieved high classification accuracy (i.e., up to 98.8%) with different designs, offering trade-offs between model accuracy and size. When deployed on mobile devices with GPU and NPU accelerators, our deep learning models achieved inference speeds up to 2.8 ms. In comparison, when deployed on our custom-designed CPU-powered smart glasses, our models yielded slower inference speeds of 1.5 s, presenting a trade-off between human-centered design and performance. Overall, the results of numerous experiments presented herein provide consistent evidence that StairNet can be an effective platform to develop and study new deep learning models for visual perception of human–robot walking environments, with an emphasis on stair recognition. This research aims to support the development of next-generation vision-based control systems for robotic prosthetic legs, exoskeletons, and other mobility assistive technologies.

Stair Recognition for Robotic Exoskeleton Control using Computer Vision and Deep Learning

Stair Recognition for Robotic Exoskeleton Control using Computer Vision and Deep Learning

StairNet: Visual Recognition of Stairs for Human-Robot Locomotion

StairNet: visual recognition of stairs for human–robot locomotion

RGB-D-based Stair Detection using Deep Learning for Autonomous Stair Climbing

StairNetV3: Depth-aware Stair Modeling using Deep Learning

sTetro-Deep Learning Powered Staircase Cleaning and Maintenance Reconfigurable Robot

Deep Learning Based Fall Recognition and Forecasting for Reconfigurable Stair-Accessing Service Robots

Deep leaning-based ultra-fast stair detection

Reinforcement Learning for Blind Stair Climbing with Legged and Wheeled-Legged Robots

An Onboard Framework for Staircases Modeling Based on Point Clouds

Staircase Localization for Autonomous Exploration in Urban Environments

Unsupervised Domain Adaptation Learning Algorithm for RGB-D Staircase Recognition

Real-Time Human Activity Recognition with IMU and Encoder Sensors in Wearable Exoskeleton Robot via Deep Learning Networks

Real-time Onboard SVM-based Human Locomotion Recognition for a Bionic Knee Exoskeleton on Different Terrains

Recognizing Sit-Stand and Stand-Sit Transitions for a Bionic Knee Exoskeleton

Deep Learning for Motion Classification in Ankle Exoskeletons Using Surface EMG and IMU Signals

Real-Time Continuous Locomotion Mode Recognition and Transition Prediction for Human With Lower Limb Exoskeleton

Personalization of Wearable Sensor-Based Joint Kinematic Estimation Using Computer Vision for Hip Exoskeleton Applications

A SE-DenseNet-LSTM model for locomotion mode recognition in lower limb exoskeleton

Multi-class SVM Based Real-Time Recognition of Sit-to-Stand and Stand-to-Sit Transitions for a Bionic Knee Exoskeleton in Transparent Mode.