Abstract:With the recent advancements in deep learning and computer vision, the AI-powered construction machine such as autonomous excavator has made significant progress. Safety is the most important section in modern construction, where construction machines are more and more automated. In this paper, we propose a vision-based excavator perception, activity analysis, and safety monitoring system. Our perception system could detect multi-class construction machines and humans in real-time while estimating the poses and actions of the excavator. Then, we present a novel safety monitoring and excavator activity analysis system based on the perception result. To evaluate the performance of our method, we collect a dataset using the Autonomous Excavator System (AES) (Zhang et al., Sci Robot 6(55):eabc3164) including multi-class of objects in different lighting conditions with human annotations. We also evaluate our method on a benchmark construction dataset. The results showed our YOLO v5 multi-class objects detection model improved inference speed by 8 times (YOLO v5 x-large) to 34 times (YOLO v5 small) compared with Faster R-CNN/YOLO v3 model (Zhang et al., In Proceedings of the 38th International Symposium on Automation and Robotics in Construction 461 (ISARC), pp. 49–56. InternationalAssociation for Automation and Robotics in Construction (IAARC), Dubai, UAE (2021). https://doi.org/10.22260/ISARC2021/0009). Furthermore, the accuracy of YOLO v5 models is improved by 2.7% (YOLO v5 x-large) while model size is reduced by 63.9% (YOLO v5 x-large) to 93.9% (YOLO v5 small). The experimental results show that the proposed action recognition approach outperforms the state-of-the-art approaches on top-1 accuracy by about 5.18%. The proposed real-time safety monitoring system is not only designed for our Autonomous Excavator System (AES) in solid waste scenes, it can also be applied to general construction scenarios.

End-to-end deep learning for reverse driving trajectory of autonomous bulldozer

Construction path tracking and pose estimation of unmanned bulldozer

An imitation from observation approach for dozing distance learning in autonomous bulldozer operation

An End-Cloud Integration Intelligent Driving System for Unmanned Bulldozer

Earthwork digital twin for teleoperation of an automated bulldozer in edge dumping

Complete coverage path planning and performance factor analysis for autonomous bulldozer

Modeling and Control of General Hydraulic Excavator for Human-in-the-loop Automation

Towards Autonomous and Optimal Excavation of Shield Machine:A Deep Reinforcement Learning-Based Approach

Deep learning technology for construction machinery and robotics

Off-road Autonomous Vehicles Traversability Analysis and Trajectory Planning Based on Deep Inverse Reinforcement Learning

Data-driven excavation trajectory planning for unmanned mining excavator

Autonomous Path Finding and Obstacle Avoidance Method for Unmanned Construction Machinery

End-to-End Autonomous Driving Through Dueling Double Deep Q-Network

An Autonomous Driving Approach Based on Trajectory Learning Using Deep Neural Networks

Construction site safety monitoring and excavator activity analysis system

Towards autonomous and optimal excavation of shield machine: a deep reinforcement learning-based approach

TOWARD AN ADAPTIVE CONTROL MODEL FOR ROBOTIC BACKHOE EXCAVATION

Deep Reinforcement Learning Based Online Lifting Path Planning for Tower Cranes in Unknown Dynamic Environments

Deep learning and control algorithms of direct perception for autonomous driving

Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues

Video surveillance-based multi-task learning with swin transformer for earthwork activity classification