Abstract:With the recent advancements in deep learning and computer vision, the AI-powered construction machine such as autonomous excavator has made significant progress. Safety is the most important section in modern construction, where construction machines are more and more automated. In this paper, we propose a vision-based excavator perception, activity analysis, and safety monitoring system. Our perception system could detect multi-class construction machines and humans in real-time while estimating the poses and actions of the excavator. Then, we present a novel safety monitoring and excavator activity analysis system based on the perception result. To evaluate the performance of our method, we collect a dataset using the Autonomous Excavator System (AES) (Zhang et al., Sci Robot 6(55):eabc3164) including multi-class of objects in different lighting conditions with human annotations. We also evaluate our method on a benchmark construction dataset. The results showed our YOLO v5 multi-class objects detection model improved inference speed by 8 times (YOLO v5 x-large) to 34 times (YOLO v5 small) compared with Faster R-CNN/YOLO v3 model (Zhang et al., In Proceedings of the 38th International Symposium on Automation and Robotics in Construction 461 (ISARC), pp. 49–56. InternationalAssociation for Automation and Robotics in Construction (IAARC), Dubai, UAE (2021). https://doi.org/10.22260/ISARC2021/0009). Furthermore, the accuracy of YOLO v5 models is improved by 2.7% (YOLO v5 x-large) while model size is reduced by 63.9% (YOLO v5 x-large) to 93.9% (YOLO v5 small). The experimental results show that the proposed action recognition approach outperforms the state-of-the-art approaches on top-1 accuracy by about 5.18%. The proposed real-time safety monitoring system is not only designed for our Autonomous Excavator System (AES) in solid waste scenes, it can also be applied to general construction scenarios.

Automatic Identification of Idling Reasons in Excavation Operations Based on Excavator–Truck Relationships

Vision-based Excavator Pose Estimation for Automatic Control

A New Measurement Method of Real-time Pose Estimation for an Automatic Hydraulic Excavator

Working Stage Identification Of Excavators Based On Control Signals Of Operating Handles

Automatic vision-based calculation of excavator earthmoving productivity using zero-shot learning activity recognition

Video-based construction vehicles detection and its application in intelligent monitoring system

Automatic excavator action recognition and localisation for untrimmed video using hybrid LSTM-Transformer networks

Automatic Mass Estimation of Construction Vehicles by Modeling Operational and Engine Data

Automating excavator productivity measurement using deep learning

Automatic Recognition of Construction Worker Activities Using Dense Trajectories

Construction site safety monitoring and excavator activity analysis system

Multi-camera vision-based productivity monitoring of earthmoving operations

Autonomous Identification and Positioning of Trucks During Collaborative Forage Harvesting

Finding Optimal Reconstruction Plans for Separating Trucks and Passenger Vehicles Systems at Urban Intersections Considering Environmental Impacts

Hydraulic Excavators Recognition Based on Inverse "V" Feature of Mechanical Arm.

Tracking and Climbing Behavior Recognition of Heavy-Duty Trucks on Roadways

Intelligent identification for working-cycle stages of excavator based on main pump pressure

Real-time Cargo Loading Identification Based on Self-powered Camera

Automated Activity Recognition of Construction Equipment Using a Data Fusion Approach

Vision-Based Excavator Detection and Tracking Using Hybrid Kinematic Shapes and Key Nodes

Dynamic Hazardous Proximity Zone Design for Excavator Based on 3D Mechanical Arm Pose Estimation Via Computer Vision