Abstract:In the complex operation scene of coal mine, due to adverse environmental factors such as uneven illumination, high dust and mixed man‐machine multi‐target, the speed and measurement accuracy of traditional visual perception model decrease sharply. In order to solve the above problems, this study proposes to build a visual information perception system for coal mine comprehensive excavation working face for edge computing terminal and combines channel pruning algorithm, knowledge extraction algorithm and TensorRT acceleration engine to realize the lightweight deployment of visual perception model on edge intelligent devices. Aiming at the problems of low detection accuracy, high computational complexity and long‐time consumption of visual perception model in a complex mining environment, this research designs a visual information perception system of coal mine comprehensive excavation working face for an edge computing terminal. Firstly, the C3‐Fast feature extraction module, spatial pyramid pooling with cross‐stage partial connection (SPPCSPC) pooling module, bi‐directional feature pyramid network and lightweight decoupled detection head are used to optimize the YOLOv5s model, so as to construct the FSBD‐YOLOv5s multi‐object detection model. Secondly, the pruning and distillation algorithm is used to lighten the FSBD‐YOLOv5s model, and the model complexity is greatly reduced while maintaining the model detection accuracy. Further, the lightweight FSBD‐YOLOv5s model is migrated and deployed to the edge computing terminal platform and the TensorRT engine is used to accelerate model inference. Finally, experiments are carried out based on the data set of the coal mine comprehensive excavation working face. The experimental results show that on the edge computing terminal platform, the parameters and computational volume of the lightweight FSBD‐YOLOv5s model are reduced by 50.8% and 34.0%, while its detection accuracy and speed reach 94.0% and 43.7 fps, which can fully satisfy the requirements of the accuracy and real‐time for the coal mine engineering applications.

Visual perception system design for rock breaking robot based on multi-sensor fusion

Evaluation of Real-Time Perception of Deformation State of Host Rocks in Coal Mine Roadways in Dusty Environment

Visual information perception system of coal mine comprehensive excavation working face for edge computing terminal

Application of Digital Image Processing Technology in Automatic Recognition Algorithm of Rock Microseismic Events in Mining Area under the Background of Mobile Communication

Multi-Sensor Fusion Tomato Picking Robot Localization and Mapping Research

Environment Perception Technology for Intelligent Robots in Complex Environments: A Review

Research on Visual–Inertial Measurement Unit Fusion Simultaneous Localization and Mapping Algorithm for Complex Terrain in Open-Pit Mines

A Novel Method for 3D Object Detection in Open-Pit Mine Based on Hybrid Solid-State LiDAR Point Cloud

A novel positioning method of anti-punching drilling robot based on the fusion of multi-IMUs and visual image

Real-time path planning of drilling robot using global and local sensor information fusion: mining-oriented validation

Vision-based Mobile Robot's Environment Outdoor Perception

Rock segmentation visual system for assisting driving in TBM construction

HVPS-DFN-DL: Intelligent Capture and Characterization of Geological Fracture Outcrops Based on a Hybrid Vision-Photogrammetric System and Discrete Fracture Network

Real-Time Dynamic Intelligent Image Recognition and Tracking System for Rockfall Disasters

Intelligent Mining Road Object Detection Based on Multiscale Feature Fusion in Multi-UAV Networks

A Detection and Tracking Method Based on Heterogeneous Multi-Sensor Fusion for Unmanned Mining Trucks

Intelligent image-based identification and 3-D reconstruction of rock fractures: Implementation and application

Vision-based size distribution analysis of rock fragments using multi-modal deep learning and interactive annotation

Coal Mine Rock Burst and Coal and Gas Outburst Perception Alarm Method Based on Visible Light Imagery

Outdoor scene understanding of mobile robot via multi-sensor information fusion

Object detection model of coal mine rescue robot based on multi-scale feature fusion