Method Towards CVPR 2021 SimLocMatch Challenge

Xiaopeng Bi,Ran Yan,Zheng Chai,Haotian Zhang,Xiao Liu

DOI: https://doi.org/10.48550/arXiv.2108.04466

2021-08-10

Computer Vision and Pattern Recognition

Abstract:This report describes Megvii-3D team's approach towards SimLocMatch Challenge @ CVPR 2021 Image Matching Workshop.

What problem does this paper attempt to address?

Method Towards CVPR 2021 Image Matching Challenge

Xiaopeng Bi,Yu Chen,Xinyang Liu,Dehao Zhang,Ran Yan,Zheng Chai,Haotian Zhang,Xiao Liu

DOI: https://doi.org/10.48550/arXiv.2108.04453

2021-08-11

Abstract:This report describes Megvii-3D team's approach towards CVPR 2021 Image Matching Workshop.

Computer Vision and Pattern Recognition,Artificial Intelligence
LocNet: Global Localization in 3D Point Clouds for Mobile Vehicles

Huan Yin,Li Tang,Xiaqing Ding,Yue Wang,Rong Xiong

DOI: https://doi.org/10.1109/ivs.2018.8500682

2017-01-01

Abstract:Global localization in 3D point clouds is a challenging problem of estimating the pose of vehicles without any prior knowledge. In this paper, a solution to this problem is presented by achieving place recognition and metric pose estimation in the global prior map. Specifically, we present a semi-handcrafted representation learning method for LiDAR point clouds using siamese LocNets, which states the place recognition problem to a similarity modeling problem. With the final learned representations by LocNet, a global localization framework with range-only observations is proposed. To demonstrate the performance and effectiveness of our global localization system, KITTI dataset is employed for comparison with other algorithms, and also on our long-time multi-session datasets for evaluation. The result shows that our system can achieve high accuracy.
3D LiDAR-Based Global Localization Using Siamese Neural Network

Huan Yin,Yue Wang,Xiaqing Ding,Li Tang,Shoudong Huang,Rong Xiong

DOI: https://doi.org/10.1109/tits.2019.2905046

IF: 8.5

2020-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:Global localization in 3D point clouds is a challenging task for mobile vehicles in outdoor scenarios, which requires the vehicle to localize itself correctly in a given map without prior knowledge of its pose. This is a critical component of autonomous vehicles or robots on the road for handling localization failures. In this paper, based on reduced dimension scan representations learned from neural networks, a solution to global localization is proposed by achieving place recognition first and then metric pose estimation in the global prior map. Specifically, we present a semi-handcrafted feature learning method for 3D Light detection and ranging (LiDAR) point clouds using artificial statistics and siamese network, which transforms the place recognition problem into a similarity modeling problem. Additionally, the sensor data using dimension reduced representations require less storage space and make the searching easier. With the learned representations by networks and the global poses, a prior map is built and used in the localization framework. In the localization step, position only observations obtained by place recognition are used in a particle filter algorithm to achieve precise pose estimation. To demonstrate the effectiveness of our place recognition and localization approach, KITTI benchmark and our multi-session datasets are employed for comparison with other geometric-based algorithms. The results show that our system can achieve both high accuracy and efficiency for long-term autonomy.
3rd Place Solution to "Google Landmark Retrieval 2020"

Ke Mei,Lei li,Jinchang Xu,Yanhua Cheng,Yugeng Lin

DOI: https://doi.org/10.48550/arXiv.2008.10480

2020-08-24

Computer Vision and Pattern Recognition

Abstract:Image retrieval is a fundamental problem in computer vision. This paper presents our 3rd place detailed solution to the Google Landmark Retrieval 2020 challenge. We focus on the exploration of data cleaning and models with metric learning. We use a data cleaning strategy based on embedding clustering. Besides, we employ a data augmentation method called Corner-Cutmix, which improves the model's ability to recognize multi-scale and occluded landmark images. We show in detail the ablation experiments and results of our method.
From Satellite to Ground: Satellite Assisted Visual Localization with Cross-view Semantic Matching

Guo Xiyue,Peng Haocheng,Hu Junjie,Bao Hujun,Zhang Guofeng

DOI: https://doi.org/10.1109/icra57147.2024.10611658

2024-01-01

Abstract:One of the key challenges of visual Simultaneous Localization and Mapping (SLAM) in large-scale environments is how to effectively use global localization to correct the cumulative errors from long-term tracking. This challenge presents itself in two main aspects: first, the difficulty for robots in revisiting previous locations to perform loop closure, and second, the considerable memory resources required to maintain point-cloud-based global maps. Recent solutions have resorted into neural networks, using satellite images as the references for ground-level localization. However, most of these methods merely provide cross-view patch-matching results, which leads to unfeasible in integration with the SLAM system. To address these issues, we present a semantic-based cross-view localization method. This approach combines semantic information with a reward and penalty mechanism, enabling us to obtain a global probability map and achieve precise 3-degree-of-freedom (3-DoF) localization. Based on that, we develop a SLAM system that capitalizes on satellite imagery for global localization. This strategy effectively bridges the gap between SLAM and real-world coordinates while also substantially reducing accumulated errors. Our experimental results demonstrate that our global localization method significantly outperforms existing satellite-based systems. Moreover, in scenarios where the robot struggles to find loop closures, employing our localization method improves the SLAM accuracy.
MegLoc: A Robust and Accurate Visual Localization Pipeline

Shuxue Peng,Zihang He,Haotian Zhang,Ran Yan,Chuting Wang,Qingtian Zhu,Xiao Liu

DOI: https://doi.org/10.48550/arXiv.2111.13063

2021-11-25

Computer Vision and Pattern Recognition

Abstract:In this paper, we present a visual localization pipeline, namely MegLoc, for robust and accurate 6-DoF pose estimation under varying scenarios, including indoor and outdoor scenes, different time across a day, different seasons across a year, and even across years. MegLoc achieves state-of-the-art results on a range of challenging datasets, including winning the Outdoor and Indoor Visual Localization Challenge of ICCV 2021 Workshop on Long-term Visual Localization under Changing Conditions, as well as the Re-localization Challenge for Autonomous Driving of ICCV 2021 Workshop on Map-based Localization for Autonomous Driving.
Multi-View Partial (MVP) Point Cloud Challenge 2021 on Completion and Registration: Methods and Results

Liang Pan,Tong Wu,Zhongang Cai,Ziwei Liu,Xumin Yu,Yongming Rao,Jiwen Lu,Jie Zhou,Mingye Xu,Xiaoyuan Luo,Kexue Fu,Peng Gao,Manning Wang,Yali Wang,Yu Qiao,Junsheng Zhou,Xin Wen,Peng Xiang,Yu-Shen Liu,Zhizhong Han,Yuanjie Yan,Junyi An,Lifa Zhu,Changwei Lin,Dongrui Liu,Xin Li,Francisco Gómez-Fernández,Qinlong Wang,Yang Yang

DOI: https://doi.org/10.48550/arXiv.2112.12053

2021-12-22

Computer Vision and Pattern Recognition

Abstract:As real-scanned point clouds are mostly partial due to occlusions and viewpoints, reconstructing complete 3D shapes based on incomplete observations becomes a fundamental problem for computer vision. With a single incomplete point cloud, it becomes the partial point cloud completion problem. Given multiple different observations, 3D reconstruction can be addressed by performing partial-to-partial point cloud registration. Recently, a large-scale Multi-View Partial (MVP) point cloud dataset has been released, which consists of over 100,000 high-quality virtual-scanned partial point clouds. Based on the MVP dataset, this paper reports methods and results in the Multi-View Partial Point Cloud Challenge 2021 on Completion and Registration. In total, 128 participants registered for the competition, and 31 teams made valid submissions. The top-ranked solutions will be analyzed, and then we will discuss future research directions.
Monocular Visual Place Recognition in LiDAR Maps via Cross-Modal State Space Model and Multi-View Matching

Gongxin Yao,Xinyang Li,Luowei Fu,Yu Pan

2024-10-09

Abstract:Achieving monocular camera localization within pre-built LiDAR maps can bypass the simultaneous mapping process of visual SLAM systems, potentially reducing the computational overhead of autonomous localization. To this end, one of the key challenges is cross-modal place recognition, which involves retrieving 3D scenes (point clouds) from a LiDAR map according to online RGB images. In this paper, we introduce an efficient framework to learn descriptors for both RGB images and point clouds. It takes visual state space model (VMamba) as the backbone and employs a pixel-view-scene joint training strategy for cross-modal contrastive learning. To address the field-of-view differences, independent descriptors are generated from multiple evenly distributed viewpoints for point clouds. A visible 3D points overlap strategy is then designed to quantify the similarity between point cloud views and RGB images for multi-view supervision. Additionally, when generating descriptors from pixel-level features using NetVLAD, we compensate for the loss of geometric information, and introduce an efficient scheme for multi-view generation. Experimental results on the KITTI and KITTI-360 datasets demonstrate the effectiveness and generalization of our method. The code will be released upon acceptance.

Computer Vision and Pattern Recognition,Robotics
Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching

Junsheng Zhou,Baorui Ma,Wenyuan Zhang,Yi Fang,Yu-Shen Liu,Zhizhong Han

2023-12-07

Abstract:Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n-Points (PnP) to estimate rigid transformation during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very different characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads to unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D elements using both voxels and points to learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented in grids, and integrate an additional point branch to regain the information lost during voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinctive patterns of cross-modality features, we design a novel loss with adaptive-weighted optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improvements over the state-of-the-art methods. The code and models are available at <a class="link-external link-https" href="https://github.com/junshengzhou/VP2P-Match" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
A Coarse-to-Fine Algorithm for Matching and Registration in 3D Cross-Source Point Clouds

Xiaoshui Huang,Jian Zhang,Qiang Wu,Lixin Fan,Chun Yuan

DOI: https://doi.org/10.1109/tcsvt.2017.2730232

IF: 5.859

2018-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:We propose an efficient method to deal with the matching and registration problem found in cross-source point clouds captured by different types of sensors. This task is especially challenging due to the presence of density variation, scale difference, a large proportion of noise and outliers, missing data, and viewpoint variation. The proposed method has two stages: in the coarse matching stage, we use the ensemble of shape functions descriptor to select potential K regions from the candidate point clouds for the target. In the fine stage, we propose a scale embedded generative Gaussian mixture models registration method to refine the results from the coarse matching stage. Following the fine stage, both the best region and accurate camera pose relationships between the candidates and target are found. We conduct experiments in which we apply the method to two applications: one is 3D object detection and localization in street-view outdoor (LiDAR/VSFM) cross-source point clouds and the other is 3D scene matching and registration in indoor (KinectFusion/VSFM) cross-source point clouds. The experiment results show that the proposed method performs well when compared with the existing methods. It also shows that the proposed method is robust under various sensing techniques, such as LiDAR, Kinect, and RGB camera.
CyberLoc: Towards Accurate Long-term Visual Localization

Liu Liu,Yukai Lin,Xiao Liang,Qichao Xu,Miao Jia,Yangdong Liu,Yuxiang Wen,Wei Luo,Jiangwei Li

DOI: https://doi.org/10.48550/arXiv.2301.02403

2023-01-06

Abstract:This technical report introduces CyberLoc, an image-based visual localization pipeline for robust and accurate long-term pose estimation under challenging conditions. The proposed method comprises four modules connected in a sequence. First, a mapping module is applied to build accurate 3D maps of the scene, one map for each reference sequence if there exist multiple reference sequences under different conditions. Second, a single-image-based localization pipeline (retrieval--matching--PnP) is performed to estimate 6-DoF camera poses for each query image, one for each 3D map. Third, a consensus set maximization module is proposed to filter out outlier 6-DoF camera poses, and outputs one 6-DoF camera pose for a query. Finally, a robust pose refinement module is proposed to optimize 6-DoF query poses, taking candidate global 6-DoF camera poses and their corresponding global 2D-3D matches, sparse 2D-2D feature matches between consecutive query images and SLAM poses of the query sequence as input. Experiments on the 4seasons dataset show that our method achieves high accuracy and robustness. In particular, our approach wins the localization challenge of ECCV 2022 workshop on Map-based Localization for Autonomous Driving (MLAD-ECCV2022).

Computer Vision and Pattern Recognition
3D Focusing-and-Matching Network for Multi-Instance Point Cloud Registration

Liyuan Zhang,Le Hui,Qi Liu,Bo Li,Yuchao Dai

2024-11-12

Abstract:Multi-instance point cloud registration aims to estimate the pose of all instances of a model point cloud in the whole scene. Existing methods all adopt the strategy of first obtaining the global correspondence and then clustering to obtain the pose of each instance. However, due to the cluttered and occluded objects in the scene, it is difficult to obtain an accurate correspondence between the model point cloud and all instances in the scene. To this end, we propose a simple yet powerful 3D focusing-and-matching network for multi-instance point cloud registration by learning the multiple pair-wise point cloud registration. Specifically, we first present a 3D multi-object focusing module to locate the center of each object and generate object proposals. By using self-attention and cross-attention to associate the model point cloud with structurally similar objects, we can locate potential matching instances by regressing object centers. Then, we propose a 3D dual masking instance matching module to estimate the pose between the model point cloud and each object proposal. It performs instance mask and overlap mask masks to accurately predict the pair-wise correspondence. Extensive experiments on two public benchmarks, Scan2CAD and ROBI, show that our method achieves a new state-of-the-art performance on the multi-instance point cloud registration task. Code is available at <a class="link-external link-https" href="https://github.com/zlynpu/3DFMNet" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
IMC 2024 Methods & Solutions Review

Shyam Gupta,Dhanisha Sharma,Songling Huang

2024-07-03

Abstract:For the past three years, Kaggle has been hosting the Image Matching Challenge, which focuses on solving a 3D image reconstruction problem using a collection of 2D images. Each year, this competition fosters the development of innovative and effective methodologies by its participants. In this paper, we introduce an advanced ensemble technique that we developed, achieving a score of 0.153449 on the private leaderboard and securing the 160th position out of over 1,000 participants. Additionally, we conduct a comprehensive review of existing methods and techniques employed by top-performing teams in the competition. Our solution, alongside the insights gathered from other leading approaches, contributes to the ongoing advancement in the field of 3D image reconstruction. This research provides valuable knowledge for future participants and researchers aiming to excel in similar image matching and reconstruction challenges.

Computer Vision and Pattern Recognition,Artificial Intelligence,Applications
Multi-Modal UAV Detection, Classification and Tracking Algorithm – Technical Report for CVPR 2024 UG2 Challenge

Tianchen Deng,Yi Zhou,Wenhua Wu,Mingrui Li,Jingwei Huang,Shuhong Liu,Yanzeng Song,Hao Zuo,Yanbo Wang,Yutao Yue,Hesheng Wang,Weidong Chen

2024-01-01

Abstract:This technical report presents the 1st winning model for UG2+, a task in CVPR 2024 UAV Tracking and Pose-Estimation Challenge. This challenge faces difficulties in drone detection, UAV-type classification and 2D/3D trajectory estimation in extreme weather conditions with multi-modal sensor information, including stereo vision, various Lidars, Radars, and audio arrays. Leveraging this information, we propose a multi-modal UAV detection, classification, and 3D tracking method for accurate UAV classification and tracking. A novel classification pipeline which incorporates sequence fusion, region of interest (ROI) cropping, and keyframe selection is proposed. Our system integrates cutting-edge classification techniques and sophisticated post-processing steps to boost accuracy and robustness. The designed pose estimation pipeline incorporates three modules: dynamic points analysis, a multi-object tracker, and trajectory completion techniques. Extensive experiments have validated the effectiveness and precision of our approach. In addition, we also propose a novel dataset pre-processing method and conduct a comprehensive ablation study for our design. We finally achieved the best performance in the classification and tracking of the MMUAD dataset. The code and configuration of our method are available at https://github.com/dtc111111/Multi-Modal-UAV.
3rd Place Solution for MOSE Track in CVPR 2024 PVUW workshop: Complex Video Object Segmentation

Xinyu Liu,Jing Zhang,Kexin Zhang,Yuting Yang,Licheng Jiao,Shuyuan Yang

2024-06-06

Abstract:Video Object Segmentation (VOS) is a vital task in computer vision, focusing on distinguishing foreground objects from the background across video frames. Our work draws inspiration from the Cutie model, and we investigate the effects of object memory, the total number of memory frames, and input resolution on segmentation performance. This report validates the effectiveness of our inference method on the coMplex video Object SEgmentation (MOSE) dataset, which features complex occlusions. Our experimental results demonstrate that our approach achieves a J\&F score of 0.8139 on the test set, securing the third position in the final ranking. These findings highlight the robustness and accuracy of our method in handling challenging VOS scenarios.

Computer Vision and Pattern Recognition,Artificial Intelligence
Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection

Benjin Zhu,Zhengkai Jiang,Xiangxin Zhou,Zeming Li,Gang Yu

DOI: https://doi.org/10.48550/arXiv.1908.09492

2019-08-26

Computer Vision and Pattern Recognition

Abstract:This report presents our method which wins the nuScenes3D Detection Challenge [17] held in Workshop on Autonomous Driving(WAD, CVPR 2019). Generally, we utilize sparse 3D convolution to extract rich semantic features, which are then fed into a class-balanced multi-head network to perform 3D object detection. To handle the severe class imbalance problem inherent in the autonomous driving scenarios, we design a class-balanced sampling and augmentation strategy to generate a more balanced data distribution. Furthermore, we propose a balanced group-ing head to boost the performance for the categories withsimilar shapes. Based on the Challenge results, our methodoutperforms the PointPillars [14] baseline by a large mar-gin across all metrics, achieving state-of-the-art detection performance on the nuScenes dataset. Code will be released at CBGS.
TM3Loc: Tightly-Coupled Monocular Map Matching for High Precision Vehicle Localization

Tuopu Wen,Kun Jiang,Benny Wijaya,Hangyu Li,Mengmeng Yang,Diange Yang

DOI: https://doi.org/10.1109/tits.2022.3176914

IF: 8.5

2022-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:Vision-based map-matching with HD map for high precision vehicle localization has gained great attention for its low-cost and ease of deployment. However, its localization performance is still unsatisfactory in accuracy and robustness in numerous real applications due to the sparsity and noise of the perceived HD map landmarks. This article proposes the tightly-coupled monocular map-matching localization algorithm (TM3Loc) for monocular-based vehicle localization. TM3Loc introduces semantic chamfer matching (SCM) to model monocular map-matching problem and combines visual features with SCM in a tightly-coupled manner. By applying the sliding window-based optimization technique, the historical visual features and HD map constraints are also introduced, such that the vehicle poses are estimated with an abundance of visual features and multi-frame HD map landmark features, rather than with single-frame HD map observations in previous works. Experiments are conducted on large scale dataset of 15 km long in total. The results show that TM3Loc is able to achieve high precision localization performance using a low-cost monocular camera, largely exceeding the performance of the previous state-of-the-art methods, thereby promoting the development of autonomous driving.
CenterAtt: Fast 2-stage Center Attention Network

Jianyun Xu,Xin Tang,Jian Dou,Xu Shu,Yushi Zhu

DOI: https://doi.org/10.48550/arXiv.2106.10493

2021-06-19

Abstract:In this technical report, we introduce the methods of HIKVISION_LiDAR_Det in the challenge of waymo open dataset real-time 3D detection. Our solution for the competition are built upon Centerpoint 3D detection framework. Several variants of CenterPoint are explored, including center attention head and feature pyramid network neck. In order to achieve real time detection, methods like batchnorm merge, half-precision floating point network and GPU-accelerated voxelization process are adopted. By using these methods, our team ranks 6th among all the methods on real-time 3D detection challenge in the waymo open dataset.

Computer Vision and Pattern Recognition
Team AcieLee: Technical Report for EPIC-SOUNDS Audio-Based Interaction Recognition Challenge 2023

Yuqi Li,Yizhi Luo,Xiaoshuai Hao,Chuanguang Yang,Zhulin An,Dantong Song,Wei Yi

2023-06-15

Abstract:In this report, we describe the technical details of our submission to the EPIC-SOUNDS Audio-Based Interaction Recognition Challenge 2023, by Team "AcieLee" (username: Yuqi\_Li). The task is to classify the audio caused by interactions between objects, or from events of the camera wearer. We conducted exhaustive experiments and found learning rate step decay, backbone frozen, label smoothing and focal loss contribute most to the performance improvement. After training, we combined multiple models from different stages and integrated them into a single model by assigning fusion weights. This proposed method allowed us to achieve 3rd place in the CVPR 2023 workshop of EPIC-SOUNDS Audio-Based Interaction Recognition Challenge.

Sound,Computer Vision and Pattern Recognition,Audio and Speech Processing
3rd Place Solution to Meta AI Video Similarity Challenge

Shuhei Yokoo,Peifei Zhu,Junki Ishikawa,Rintaro Hasegawa

2023-05-18

Abstract:This paper presents our 3rd place solution in both Descriptor Track and Matching Track of the Meta AI Video Similarity Challenge (VSC2022), a competition aimed at detecting video copies. Our approach builds upon existing image copy detection techniques and incorporates several strategies to exploit on the properties of video data, resulting in a simple yet powerful solution. By employing our proposed method, we achieved substantial improvements in accuracy compared to the baseline results (Descriptor Track: 38% improvement, Matching Track: 60% improvement). Our code is publicly available here: <a class="link-external link-https" href="https://github.com/line/Meta-AI-Video-Similarity-Challenge-3rd-Place-Solution" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

Method Towards CVPR 2021 SimLocMatch Challenge

Method Towards CVPR 2021 Image Matching Challenge

LocNet: Global Localization in 3D Point Clouds for Mobile Vehicles

3D LiDAR-Based Global Localization Using Siamese Neural Network

3rd Place Solution to "Google Landmark Retrieval 2020"

From Satellite to Ground: Satellite Assisted Visual Localization with Cross-view Semantic Matching

MegLoc: A Robust and Accurate Visual Localization Pipeline

Multi-View Partial (MVP) Point Cloud Challenge 2021 on Completion and Registration: Methods and Results

Monocular Visual Place Recognition in LiDAR Maps via Cross-Modal State Space Model and Multi-View Matching

Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching

A Coarse-to-Fine Algorithm for Matching and Registration in 3D Cross-Source Point Clouds

CyberLoc: Towards Accurate Long-term Visual Localization

3D Focusing-and-Matching Network for Multi-Instance Point Cloud Registration

IMC 2024 Methods & Solutions Review

Multi-Modal UAV Detection, Classification and Tracking Algorithm – Technical Report for CVPR 2024 UG2 Challenge

3rd Place Solution for MOSE Track in CVPR 2024 PVUW workshop: Complex Video Object Segmentation

Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection

TM3Loc: Tightly-Coupled Monocular Map Matching for High Precision Vehicle Localization

CenterAtt: Fast 2-stage Center Attention Network

Team AcieLee: Technical Report for EPIC-SOUNDS Audio-Based Interaction Recognition Challenge 2023

3rd Place Solution to Meta AI Video Similarity Challenge