Abstract:With the recent advancements in deep learning and computer vision, the AI-powered construction machine such as autonomous excavator has made significant progress. Safety is the most important section in modern construction, where construction machines are more and more automated. In this paper, we propose a vision-based excavator perception, activity analysis, and safety monitoring system. Our perception system could detect multi-class construction machines and humans in real-time while estimating the poses and actions of the excavator. Then, we present a novel safety monitoring and excavator activity analysis system based on the perception result. To evaluate the performance of our method, we collect a dataset using the Autonomous Excavator System (AES) (Zhang et al., Sci Robot 6(55):eabc3164) including multi-class of objects in different lighting conditions with human annotations. We also evaluate our method on a benchmark construction dataset. The results showed our YOLO v5 multi-class objects detection model improved inference speed by 8 times (YOLO v5 x-large) to 34 times (YOLO v5 small) compared with Faster R-CNN/YOLO v3 model (Zhang et al., In Proceedings of the 38th International Symposium on Automation and Robotics in Construction 461 (ISARC), pp. 49–56. InternationalAssociation for Automation and Robotics in Construction (IAARC), Dubai, UAE (2021). https://doi.org/10.22260/ISARC2021/0009). Furthermore, the accuracy of YOLO v5 models is improved by 2.7% (YOLO v5 x-large) while model size is reduced by 63.9% (YOLO v5 x-large) to 93.9% (YOLO v5 small). The experimental results show that the proposed action recognition approach outperforms the state-of-the-art approaches on top-1 accuracy by about 5.18%. The proposed real-time safety monitoring system is not only designed for our Autonomous Excavator System (AES) in solid waste scenes, it can also be applied to general construction scenarios.

Automatic Recognition of Construction Worker Activities Using Dense Trajectories

Construction Motion Data Library: an Integrated Motion Dataset for On-Site Activity Recognition

Vision-Based Construction Worker Activity Analysis Informed by Body Posture

Deep learning-based recognition of construction activities in real construction site environment

Activity Recognition Using Dense Long-Duration Trajectories

Automated Activity Recognition of Construction Equipment Using a Data Fusion Approach

Recognizing Construction Worker Activities Based on Accelerometers

Construction worker detection in video frames for initializing vision trackers

Deep learning-based activity-aware 3D human motion trajectory prediction in construction

Dual attention-based deep learning for construction equipment activity recognition considering transition activities and imbalanced dataset

Human Activity Recognition based on Dynamic Spatio-Temporal Relations

Recognizing workers' construction activities on a reinforcement processing area through the position relationship of objects detected by faster R-CNN

Automatic vision-based calculation of excavator earthmoving productivity using zero-shot learning activity recognition

Development of an action classification method for construction sites combining pose assessment and object proximity evaluation

Multi-Modal Recognition of Worker Activity for Human-Centered Intelligent Manufacturing

Personnel tracking on construction sites using video cameras

Intelligent Technologies for Construction Machinery Using Data-Driven Methods

Construction site safety monitoring and excavator activity analysis system

Tracking indoor construction progress by deep-learning-based analysis of site surveillance video

Robust Activity Recognition for Adaptive Worker-Robot Interaction using Transfer Learning