An autonomous excavator system for material loading tasks

Liangjun Zhang,Jinxin Zhao,Pinxin Long,Liyang Wang,Lingfeng Qian,Feixiang Lu,Xibin Song,Dinesh Manocha

DOI: https://doi.org/10.1126/scirobotics.abc3164

IF: 25

2021-06-23

Science Robotics

Abstract:An autonomous excavator system (AES) operates in the real world over long time periods.

robotics

What problem does this paper attempt to address?

AES: Autonomous Excavator System for Real-World and Hazardous Environments

Jinxin Zhao,Pinxin Long,Liyang Wang,Lingfeng Qian,Feixiang Lu,Xibin Song,Dinesh Manocha,Liangjun Zhang

DOI: https://doi.org/10.48550/arXiv.2011.04848

IF: 3.7

2020-11-10

Robotics

Abstract:Excavators are widely used for material-handling applications in unstructured environments, including mining and construction. The size of the global market of excavators is 44.12 Billion USD in 2018 and is predicted to grow to 63.14 Billion USD by 2026. Operating excavators in a real-world environment can be challenging due to extreme conditions and rock sliding, ground collapse, or exceeding dust. Multiple fatalities and injuries occur each year during excavations. An autonomous excavator that can substitute human operators in these hazardous environments would substantially lower the number of injuries and can improve the overall productivity.
Vision-based Excavator Pose Estimation for Automatic Control

Guangxu Liu,Qingfeng Wang,Tao Wang,Bingcheng Li,Xiangshuo Xi

DOI: https://doi.org/10.1016/j.autcon.2023.105162

IF: 10.3

2024-01-01

Automation in Construction

Abstract:Hydraulic excavators, widely employed in harsh environments, have garnered increased attention in recent years from manufacturers and researchers for automatic operation, particularly through vision-based manipulator pose measurement. This paper utilized a mixed review method to explore the vision-based measurements as well as other applications. A quantitative analysis of computer vision (CV) applications on excavators was performed through a literature search, leading to the classification of these applications into three categories based on keywords. Subsequently, a comprehensive literature review identified four key requirements and challenges for vision-based measurement in the context of automatic control, followed by a discussion on the future prospects of marker-based and deep learning-based manipulator pose measurement for automatic control. This paper offers an in-depth examination of CV applications in excavators, emphasizing the pertinent challenges and forthcoming trends in vision-based manipulator pose measurement for automatic control.
Modeling and Control of General Hydraulic Excavator for Human-in-the-loop Automation

Guangda Chen,Yinghao Gan,Jiayi Chen,Shuanwu Shi,Wei Chen,Yingfeng Chen,Rong Xiong,Changjie Fan

DOI: https://doi.org/10.1109/ictai59109.2023.00110

2023-01-01

Abstract:As labor shortages and safety regulations become more prominent, the need for human-in-the-loop automation of excavators is increasing. To meet this demand, we have developed a comprehensive modeling method for the excavator arm using nonlinear optimization approaches, including a simplified model that maps the task space to the joint space, as well as an equivalent model that maps the joint space to the actuator space. These models were then used to build a feedforward-PID joint velocity controller and a joint trajectory controller combined with position feedback, which forms the core of our proposed semi-automatic control system for the excavator arm. Our deployment scheme is simple and efficient, and has been deployed on two excavators of different makes and sizes. Experiments show that our deployment scheme performs well on both excavators, with an average error of 0.05 rad/s for the velocity controller and less than 5 cm for the trajectory controller. Using our semi-automatic system, we have completed demonstration experiments for precise digging and grading operations. A demonstration video can be found at https://youtu.be/N6I0WZGSF68.
ExACT: An End-to-End Autonomous Excavator System Using Action Chunking With Transformers

Liangliang Chen,Shiyu Jin,Haoyu Wang,Liangjun Zhang

2024-05-09

Abstract:Excavators are crucial for diverse tasks such as construction and mining, while autonomous excavator systems enhance safety and efficiency, address labor shortages, and improve human working conditions. Different from the existing modularized approaches, this paper introduces ExACT, an end-to-end autonomous excavator system that processes raw LiDAR, camera data, and joint positions to control excavator valves directly. Utilizing the Action Chunking with Transformers (ACT) architecture, ExACT employs imitation learning to take observations from multi-modal sensors as inputs and generate actionable sequences. In our experiment, we build a simulator based on the captured real-world data to model the relations between excavator valve states and joint velocities. With a few human-operated demonstration data trajectories, ExACT demonstrates the capability of completing different excavation tasks, including reaching, digging and dumping through imitation learning in validations with the simulator. To the best of our knowledge, ExACT represents the first instance towards building an end-to-end autonomous excavator system via imitation learning methods with a minimal set of human demonstrations. The video about this work can be accessed at

Robotics,Artificial Intelligence,Machine Learning
HEAP -- The autonomous walking excavator

Dominic Jud,Simon Kerscher,Martin Wermelinger,Edo Jelavic,Pascal Egli,Philipp Leemann,Gabriel Hottiger,Marco Hutter

DOI: https://doi.org/10.1016/j.autcon.2021.103783

2021-06-09

Abstract:The demand and the potential for automation in the construction sector is unmatched, particularly for increasing environmental sustainability, improving worker safety and reducing labor shortages. We have developed an autonomous walking excavator - based one of the most versatile machines found on construction sites - as one way to begin fulfilling this potential. This article describes the process of converting an off-the-shelf construction machine into an autonomous robotic system. First we outline the necessary sensing equipment for full autonomy and the novel actuation of the legs, and compare three different complementary actuation principles for the excavator's arm. Second, we solve the state estimation problem for a general wheeled-legged robot. Beside kinematic measurements, it includes GNSS-RTK, to absolutely reference the machine on a construction site. Third, we developed individual controllers for driving, chassis balancing and arm motions allowing for fully autonomous operation. Lastly, we highlight the machine's potential in four different real-world applications, e.g. autonomous trench digging, autonomous assembly of dry stone walls, autonomous forestry work and semi-autonomous teleoperation. On top, we also share some development insights and possible future research directions.

Robotics
Towards Autonomous Excavation Planning

Lorenzo Terenzi,Marco Hutter

2023-08-22

Abstract:Excavation plans are crucial in construction projects, dictating the dirt disposal strategy and excavation sequence based on the final geometry and machinery available. While most construction processes rely heavily on coarse sequence planning and local execution planning driven by human expertise and intuition, fully automated planning tools are notably absent from the industry. This paper introduces a fully autonomous excavation planning system. Initially, the site is mapped, followed by user selection of the desired excavation geometry. The system then invokes a global planner to determine the sequence of poses for the excavator, ensuring complete site coverage. For each pose, a local excavation planner decides how to move the soil around the machine, and a digging planner subsequently dictates the sequence of digging trajectories to complete a patch. We showcased our system by autonomously excavating the largest pit documented so far, achieving an average digging cycle time of roughly 30 seconds, comparable to the one of a human operator.

Robotics
Construction site safety monitoring and excavator activity analysis system

Sibo Zhang,Liangjun Zhang

DOI: https://doi.org/10.1007/s41693-022-00077-0

2022-06-01

Construction Robotics

Abstract:With the recent advancements in deep learning and computer vision, the AI-powered construction machine such as autonomous excavator has made significant progress. Safety is the most important section in modern construction, where construction machines are more and more automated. In this paper, we propose a vision-based excavator perception, activity analysis, and safety monitoring system. Our perception system could detect multi-class construction machines and humans in real-time while estimating the poses and actions of the excavator. Then, we present a novel safety monitoring and excavator activity analysis system based on the perception result. To evaluate the performance of our method, we collect a dataset using the Autonomous Excavator System (AES) (Zhang et al., Sci Robot 6(55):eabc3164) including multi-class of objects in different lighting conditions with human annotations. We also evaluate our method on a benchmark construction dataset. The results showed our YOLO v5 multi-class objects detection model improved inference speed by 8 times (YOLO v5 x-large) to 34 times (YOLO v5 small) compared with Faster R-CNN/YOLO v3 model (Zhang et al., In Proceedings of the 38th International Symposium on Automation and Robotics in Construction 461 (ISARC), pp. 49–56. InternationalAssociation for Automation and Robotics in Construction (IAARC), Dubai, UAE (2021). https://doi.org/10.22260/ISARC2021/0009). Furthermore, the accuracy of YOLO v5 models is improved by 2.7% (YOLO v5 x-large) while model size is reduced by 63.9% (YOLO v5 x-large) to 93.9% (YOLO v5 small). The experimental results show that the proposed action recognition approach outperforms the state-of-the-art approaches on top-1 accuracy by about 5.18%. The proposed real-time safety monitoring system is not only designed for our Autonomous Excavator System (AES) in solid waste scenes, it can also be applied to general construction scenarios.
Towards Autonomous and Optimal Excavation of Shield Machine:A Deep Reinforcement Learning-Based Approach

Zhang Ya-kun,Gong Guo-fang,Yang Hua-yong,Chen Yu-xi,Chen Geng-lin

DOI: https://doi.org/10.1631/jzus.a2100325

2022-01-01

Journal of Zhejiang University SCIENCE A

Abstract:Autonomous excavation operation is a major trend in the development of a new generation of intelligent tunnel boring machines (TBMs). However, existing technologies are limited to supervised machine learning and static optimization, which cannot outperform human operation and deal with ever changing geological conditions and the long-term performance measure. The aim of this study is to resolve the problem of dynamic optimization of the shield excavation performance, as well as to achieve autonomous optimal excavation. In this study, a novel autonomous optimal excavation approach that integrates deep reinforcement learning and optimal control is proposed for shield machines. Based on a first-principles analysis of the machine-ground interaction dynamics of the excavation process, a deep neural network model is developed using construction field data consisting of 1.1 million samples. The multi-system coupling mechanism is revealed by establishing an overall system model. Based on the overall system analysis, the autonomous optimal excavation problem is decomposed into a multi-objective dynamic optimization problem and an optimal control problem. Subsequently, a dimensionless multi-objective comprehensive excavation performance measure is proposed. A deep reinforcement learning method is used to solve for the optimal action sequence trajectory, and optimal closed-loop feedback controllers are designed to achieve accurate execution. The performance of the proposed approach is compared to that of human operation by using the construction field data. The simulation results show that the proposed approach not only has the potential to replace human operation but also can significantly improve the comprehensive excavation performance.
Design and Development of Intelligent Unmanned Construction System for Open-Pit Mine

Kui Liu,Bin Mei,Qing Li

DOI: https://doi.org/10.1088/1742-6596/2741/1/012005

2024-01-01

Journal of Physics Conference Series

Abstract:Abstract China’s huge mineral outputs support the national economy and people’s livelihood. However, the mining operation is rough and the safety accident happens frequently. Consequently, the demand for unmanned mining operations is urgent, which is of great significance to safe production. Facing the complex operation scenarios of open-pit mines, an unmanned construction system framework is proposed based on intelligent construction machinery. On this basis, an unmanned excavation subsystem is designed and developed based on the hydraulic excavator, and an unmanned transportation subsystem is designed and developed based on the mining truck. Equipped with advanced sensors and controllers, the unmanned excavation subsystem and the unmanned transportation subsystem possess sensing, decision-making, and control abilities by using AI algorithms. The experiment of the unmanned construction systems is successfully conducted on the open-pit mine test site. This study provides equipment and technical support for improving the intelligent level of mine construction and ensuring safe production for the open-pit mine.
Digital twin-based excavation trajectory generation of Uncrewed excavators for autonomous mining

Tao Fu,Tianci Zhang,Yilin Lv,Xueguan Song,Guang Li,Haifeng Yue

DOI: https://doi.org/10.1016/j.autcon.2023.104855

IF: 10.3

2023-04-12

Automation in Construction

Abstract:As modern large-capacity excavators, cable shovels provide payload capacities of more than 100-tons per scoop. The presence of a high dynamic payload combined with the self-weight may result in severe stress loading of the whole shovel front-end components during excavation. Structural performance analysis based on numerical simulations typically demands a prohibitively heavy computation, so conventional optimization-based excavation trajectory generation methods generally do not consider structural performance constraints, which may cause sudden component failure or unanticipated costly downtime. Herein, a shape–performance integrated digital twin was constructed to analyze the shape and structural performance of an uncrewed cable shovel (UCS) in real time, which was further used to support optimal excavation trajectory generation. The proposed procedure presents salient features that the time-consuming structural performance response was taken as an additional constraint, which could effectively prevent the sudden structural failures. The effectiveness and applicability were demonstrated via field tests with a UCS prototype (1:7 scale model of the WK-55 type).

construction & building technology,engineering, civil
Research and design of excavation face remote reproduction and remote control systems

C. Liu,J. G. Jiang,W. H. Qu,Z. Z. Zhou,J. J. Li,H. Liu,S. Ye,S. Y. Yao

DOI: https://doi.org/10.1201/9781315212876-60

2017-01-01

Abstract:This paper referred to the National Key Basic Research Program of China (grant no. 2014CB046306), Key Fundamental Research on the Unmanned Mining Equipment in Deep Dangerous Coal Bed, as the topic background and showed the unmanned fully mechanized excavation face for deep coal seam proposed for higher requirements related to efficiency, automation, and intelligentization of the roadheader. Research and development on fully mechanized excavation face remote reproduction and remote control systems is discussed in this paper. The systems comprise the airborne control system, the industrial supervisory control computer, the Linux operating systems and the CAN bus communication. By the optical fiber and the airborne computer, the supervisory control software platform is used to transmit data and control remotely the openness of signals for the roadheader; thus, it has extensive application prospect, which provides technical support for unmanned excavation equipment, which really achieve development of fully mechanized excavation face machinery aimed at the direction of upsizing, intelligentization, and unmanned property.
An All-terrain-adaptive Off-road Environment-detecting Machine

Hai Wang,Haitao Zhang,Yong Lu

DOI: https://doi.org/10.7690/bgzdh.2016.10.017

2016-01-01

Abstract:Considering the difficulty of artificial detection in the harsh environment after earthquakes, mining accidents and other disasters, the paper discusses the design of an all-terrain-adaptive environment-detecting machine. From the chassis part, six degrees of freedom manipulator, drive system and remote monitoring for the overall design, and through the special design of the chassis to ensure the needs of climbing ability, construct an adaptive off-road machine. The test results show that the machine has good off-road shock absorption performance, and can achieve electronically controlled continuously variable transmission.
TNES: terrain traversability mapping, navigation and excavation system for autonomous excavators on worksite

Tianrui Guan,Zhenpeng He,Ruitao Song,Liangjun Zhang

DOI: https://doi.org/10.1007/s10514-023-10113-9

IF: 3.255

2023-07-05

Autonomous Robots

Abstract:We present a terrain traversability mapping and navigation system (TNS) for autonomous excavator applications in an unstructured environment. We use an efficient approach to extract terrain features from RGB images and 3D point clouds and incorporate them into a global map for planning and navigation. Our system can adapt to changing environments and update the terrain information in real-time. Moreover, we present a novel dataset, the Complex Worksite Terrain dataset, which consists of RGB images from construction sites with seven categories based on navigability. Our novel algorithms improve the mapping accuracy over previous methods by 4.17–30.48 and reduce MSE on the traversability map by 13.8–71.4 . We have combined our mapping approach with planning and control modules in an autonomous excavator navigation system and observe improvement in the overall success rate. Based on TNS, we demonstrate the first autonomous excavator that can navigate through unstructured environments consisting of deep pits, steep hills, rock piles, and other complex terrain features. In addition, we combine the proposed TNS with the autonomous excavation system (AES), and deploy the new pipeline, TNES, on a more complex construction site. With minimum human intervention, we demonstrate autonomous navigation capability with excavation tasks.

robotics,computer science, artificial intelligence
Research on Trajectory Planning and Autodig of Hydraulic Excavator

Bin Zhang,Shuang Wang,Yuting Liu,Huayong Yang

DOI: https://doi.org/10.1155/2017/7139858

IF: 1.43

2017-01-01

Mathematical Problems in Engineering

Abstract:As the advances in computer control technology keep emerging, robotic hydraulic excavator becomes imperative. It can improve excavation accuracy and greatly reduce the operator's labor intensity. The 12-ton backhoe bucket excavator has been utilized in this research work where this type of excavator is commonly used in engineering work. The kinematics model of operation device (boom, arm, bucket, and swing) in excavator is established in both Denavit-Hartenberg coordinates for easy programming and geometric space for avoiding blind spot. The control approach is based on trajectory tracing method with displacements and velocities feedbacks. The trajectory planning and auto dig programis written by Visual C++. By setting the bucket teeth's trajectory, the program can automatically plan the velocity and acceleration of each hydraulic cylinder and motor. The results are displayed through a 3D entity simulation environment which can present real-time movements of excavator kinematics. Object-Oriented Graphics Rendering Engine and skeletal animation are used to give accurate parametric control and feedback. The simulation result shows that a stable linear auto dig can be achieved. The errors between trajectory planning command and simulation model are analyzed.
A Real-Time Simulation System For Excavator Controller

Wen Gong,Qingfeng Wang

DOI: https://doi.org/10.1115/1.802977.paper58

2009-01-01

Abstract:In order to supply a reliable and low-cost method for testing excavator's controllers and evaluating the performance of strategies, an excavator real-time simulation system has been developed. The hardware-in-the-loop system, including dynamic model of the arms, dynamic model of the main hydraulic system and an 3D real-time display subsystem, is presented in this paper. The simulation results are also described at the end of the paper.
Autonomous Soil Vision Scanning System for Intelligent Subgrade Compaction

Xuefei Wang,Tingkai Wang,Jianmin Zhang,Guowei Ma

DOI: https://doi.org/10.1016/j.autcon.2023.105242

IF: 10.3

2023-01-01

Automation in Construction

Abstract:The wide application of intelligent compaction in subgrade construction is limited by the evaluation accuracy. The current practice assumes the compacted site as a homogeneous soil condition, ignoring the influence of soil property. This paper describes an advanced autonomous soil scanning system based on the computer-vision approach to recognize the soil gradation in real-time. A hybrid intelligent model is developed through a series of field compaction tests, integrating Deeplabv3+ and XGBoost algorithms. The proposed method is effective in segmenting and quantifying the large particles while the fine-grained soil is completed by the prediction model. The scanned soil information correlates well with the intelligent compaction indicators, providing essential parameters for the accurate assessment of subgrade compaction quality. The soil vision scanning system is combined with the intelligent compaction system to enable the roller autonomous learning of the whole construction area.
High-Precision Positioning, Perception and Safe Navigation for Automated Heavy-duty Mining Trucks

Long Chen,Yuchen Li,Luxi Li,Shuangying Qi,Jian Zhou,Youchen Tang,Jianjian Yang,Jingmin Xin

DOI: https://doi.org/10.1109/tiv.2024.3375273

IF: 8.2

2024-01-01

IEEE Transactions on Intelligent Vehicles

Abstract:Autonomous driving technology has achieved significant breakthroughs in open scenarios, enabling the deployment of excellent positioning, detection, and navigation algorithms on passenger vehicles. However, there has been limited research attention devoted to autonomous driving for specialized vehicles in non-open scenarios. This manuscript introduces a perception system designed for heavy-duty mining transportation trucks operating in open-pit mines, which are typical of non-open scenarios. The system comprises four independent algorithms: high-precision fusion positioning, multi-task 2D detection, 9 Degrees of Freedom (9 DoF) 3D head, and autonomous navigation technology. Experimental verification demonstrates the effectiveness of these methods in addressing the challenges posed by mining environments, ultimately leading to enhanced safety and efficiency for trucks. This research outcome, through the comprehensive examination of positioning, detection, and navigation, aims to address the challenges encountered by mining trucks during operations. Its significance lies in enhancing automation levels in mining scenarios.
TNS: Terrain Traversability Mapping and Navigation System for Autonomous Excavators

Tianrui Guan,Zhenpeng He,Ruitao Song,Dinesh Manocha,Liangjun Zhang

DOI: https://doi.org/10.48550/arXiv.2109.06250

IF: 3.7

2021-09-13

Robotics

Abstract:We present a terrain traversability mapping and navigation system (TNS) for autonomous excavator applications in an unstructured environment. We use an efficient approach to extract terrain features from RGB images and 3D point clouds and incorporate them into a global map for planning and navigation. Our system can adapt to changing environments and update the terrain information in real-time. Moreover, we present a novel dataset, the Complex Worksite Terrain (CWT) dataset, which consists of RGB images from construction sites with seven categories based on navigability. Our novel algorithms improve the mapping accuracy over previous SOTA methods by 4.17-30.48% and reduce MSE on the traversability map by 13.8-71.4%. We have combined our mapping approach with planning and control modules in an autonomous excavator navigation system and observe 49.3% improvement in the overall success rate. Based on TNS, we demonstrate the first autonomous excavator that can navigate through unstructured environments consisting of deep pits, steep hills, rock piles, and other complex terrain features.
E-GTN: Advanced Terrain Sensing Framework for Enhancing Intelligent Decision Making of Excavators

Qianyou Zhao,Le Gao,Duidi Wu,Xinyao Meng,Jin Qi,Jie Hu

DOI: https://doi.org/10.3390/app14166974

2024-01-01

Applied Sciences

Abstract:The shift towards autonomous excavators in construction and mining is a significant leap towards enhancing operational efficiency and ensuring worker safety. However, it presents challenges such as the need for sophisticated decision-making and environmental perception due to complex terrains and diverse conditions. Our study introduces the E-GTN framework, a novel approach tailored for autonomous excavation that leverages advanced multisensor fusion and a custom-designed convolutional neural network to address these challenges. Results demonstrate that GridNet effectively processes grid data, enabling the reinforcement learning algorithm to make informed decisions, thereby ensuring efficient and intelligent autonomous excavator performance. The study concludes that the E-GTN framework offers a robust solution for the challenges in unmanned excavator operations, providing a valuable platform for future advancements in the field.
Scenario Engineering for Autonomous Transportation: A New Stage in Open-Pit Mines

Siyu Teng,Xuan Li,Yuchen Li,Lingxi Li,Zhe Xuanyuan,Yunfeng Ai,Long Chen

DOI: https://doi.org/10.1109/tiv.2024.3373495

IF: 8.2

2024-01-01

IEEE Transactions on Intelligent Vehicles

Abstract:In recent years, open-pit mining has seen significant advancement, the cooperative operation of various specialized machinery substantially enhancing the efficiency of mineral extraction. However, the harsh environment and complex conditions in open-pit mines present substantial challenges for the implementation of autonomous transportation systems. This research introduces a novel paradigm that integrates Scenario Engineering (SE) with autonomous transportation systems to significantly improve the trustworthiness, robustness, and efficiency in open-pit mines by incorporating the four key components of SE, including Scenario Feature Extractor, Intelligence and Index, Calibration and Certification, and Verification and Validation. This paradigm has been validated in two famous open-pit mines, the experiment results demonstrate marked improvements in robustness, trustworthiness, and efficiency. By enhancing the capacity, scalability, and diversity of autonomous transportation, this paradigm fosters the integration of SE and parallel driving and finally propels the achievement of the 6S objectives.

English Else

An autonomous excavator system for material loading tasks

AES: Autonomous Excavator System for Real-World and Hazardous Environments

Vision-based Excavator Pose Estimation for Automatic Control

Modeling and Control of General Hydraulic Excavator for Human-in-the-loop Automation

ExACT: An End-to-End Autonomous Excavator System Using Action Chunking With Transformers

HEAP -- The autonomous walking excavator

Towards Autonomous Excavation Planning

Construction site safety monitoring and excavator activity analysis system

Towards Autonomous and Optimal Excavation of Shield Machine:A Deep Reinforcement Learning-Based Approach

Design and Development of Intelligent Unmanned Construction System for Open-Pit Mine

Digital twin-based excavation trajectory generation of Uncrewed excavators for autonomous mining

Research and design of excavation face remote reproduction and remote control systems

An All-terrain-adaptive Off-road Environment-detecting Machine

TNES: terrain traversability mapping, navigation and excavation system for autonomous excavators on worksite

Research on Trajectory Planning and Autodig of Hydraulic Excavator

A Real-Time Simulation System For Excavator Controller

Autonomous Soil Vision Scanning System for Intelligent Subgrade Compaction

High-Precision Positioning, Perception and Safe Navigation for Automated Heavy-duty Mining Trucks

TNS: Terrain Traversability Mapping and Navigation System for Autonomous Excavators

E-GTN: Advanced Terrain Sensing Framework for Enhancing Intelligent Decision Making of Excavators

Scenario Engineering for Autonomous Transportation: A New Stage in Open-Pit Mines