Abstract:On January 3, 2019, the Chang'e-4 (CE-4) probe successfully landed in the Von Kármán crater inside the South Pole-Aitken (SPA) basin. With the support of a relay communication satellite "Queqiao" launched in 2018 and located at the Earth-Moon L2 liberation point, the lander and the Yutu-2 rover carried out in-situ exploration and patrol surveys, respectively, and were able to make a series of important scientific discoveries. Owing to the complexity and unpredictability of the lunar surface, teleoperation has become the most important control method for the operation of the rover. Computer vision is an important technology to support the teleoperation of the rover. During the powered descent stage and lunar surface exploration, teleoperation based on computer vision can effectively overcome many technical challenges, such as fast positioning of the landing point, high-resolution seamless mapping of the landing site, localization of the rover in the complex environment on the lunar surface, terrain reconstruction, and path planning. All these processes helped achieve the first soft landing, roving, and in-situ exploration on the lunar farside. This paper presents a high-precision positioning technology and positioning results of the landing point based on multi-source data, including orbital images and CE-4 descent images. The method and its results have been successfully applied in an actual engineering mission for the first time in China, providing important support for the topographical analysis of the landing site and mission planning for subsequent teleoperations. After landing, a 0.03 m resolution DOM was generated using the descent images and was used as one of the base maps for the overall rover path planning. Before each movement, the Yutu-2 rover controlled its hazard avoidance cameras (Hazcam), navigation cameras (Navcam), and panoramic cameras (Pancam) to capture stereo images of the lunar surface at different angles. Local digital elevation models (DEMs) with a 0.02 m resolution were routinely produced at each waypoint using the Navcam and Hazcam images. These DEMs were then used to design an obstacle recognition method and establish a model for calculating the slope, aspect, roughness, and visibility. Finally, in combination with the Yutu-2 rover mobility characteristics, a comprehensive cost map for path search was generated.By the end of the first 12 lunar days, the Yutu-2 rover has been working on the lunar farside for more than 300 days, greatly exceeding the projected service life. The rover was able to overcome the complex terrain on the lunar farside, and travelled a total distance of more than 300 m, achieving the "double three hundred" breakthrough. In future manned lunar landing and exploration of Mars by China, computer vision will play an integral role to support science target selection and scientific investigations, and will become an extremely important core technology for various engineering tasks.

A Preliminary Study on Intelligent Perception of Extraterrestrial Body Surface Based on Deep Learning and Transfer Learning

VISION BASED OBSTACLE DETECTION USING ROVER STEREO IMAGES

Deep Multi-Layer Perception Based Terrain Classification For Planetary Exploration Rovers

Adaptive Cooperative Detection Method for Unmanned Planetary Vehicles Based on Deep Reinforcement Learning

Fast Inspection and Accurate Recognition of Target Objects for Astronaut Robots Through Deep Learning

Perception and Scene Modeling of Complex Terrain Information in Patrol and Exploration Tasks

Terrain Classification Using Mars Raw Images Based on Deep Learning Algorithms with Application to Wheeled Planetary Rovers

Deep Learning based Semantic Segmentation for Mars Rover Terrain Classification

Tracking In-Cabin Astronauts Using Deep Learning and Head Motion Clues.

Residual Grounding Transformer Network for Terrain Recognition on the Lunar Surface.

Dynamic Obstacle Detection and Representation Approach for Unmanned Vehicles Based on Laser Sensor

Three-Dimensional Vibration-Based Terrain Classification for Mobile Robots

Mobile-DeepRFB: A Lightweight Terrain Classifier for Automatic Mars Rover Navigation

Automatic Rocks Segmentation Based on Deep Learning for Planetary Rover Images

COMPUTER VISION IN THE TELEOPERATION OF THE YUTU-2 ROVER

Multi-objective Recognition Based on Deep Learning

Planetary Mapping Using Deep Learning: A Method to Evaluate Feature Identification Confidence Applied to Habitats in Mars-Analog Terrain

Convolutional Neural Network Based Obstacle Detection for Unmanned Surface Vehicle.

Autonomous Martian rock image classification based on transfer deep learning methods

Obstacle visual sensing based on deep learning for low-altitude small unmanned aerial vehicles

Dense Feature Matching for Hazard Detection and Avoidance Using Machine Learning in Complex Unstructured Scenarios