Abstract:Recently, direct visual localization with convolutional neural networks has attracted researchers' attention with achieving an end-to-end process. However, on the one side, the lack of using 3D information leads to imprecise accuracy. Meanwhile, the single input image confuses the relocalization in the scenes that keep similar views at different positions. On the other side, the relocalization problem in variable or dynamic scenes is still challenging. Concentrating on these concerns, we propose two multitask relocalization networks called MMLNet and MMLNet+ for obtaining the 6-DoF camera pose in static, variable and dynamic scenes. Firstly, addressing the dataset lack of variable scenes, we construct a variable scene dataset with a semiautomatic process combining SFM and MVS algorithms with a few manual labels. Based on the process, three scenes covering an office, a bedroom and a sitting room are gathered and generated. Secondly, to enhance the perception between 2D images and 3D poses, we design a multitask network called MMLNet that regresses both camera pose and scene point cloud. Meanwhile, the Chamfer distance is joined into the original pose loss to optimize MMLNet. Moreover, MMLNet learns the pose trajectory feature by using LSTM layers to the additional pose array input, which meanwhile breaks through the limitation of single image input. Based on the MMLNet, aiming at dynamic and variable scenes, MMLNet+ outputs the auxiliary segmentation branch that distinguishes fixed, changeable or dynamic parts of the input image. Furthermore, we define the feature fusion block to implement the feature sharing among three tasks, further promoting the performance in dynamic and variable environments. Finally, experiments on static, dynamic and our constructed variable datasets demonstrate state-of-the-art relocalization performances of MMLNet and MMLNet+. Simultaneously, the positive effects of the pose learning part, reconstruction branch and segmentation task are also illustrated.

A new approach to train convolutional neural networks for real-time 6-DOF camera relocalization

A New Approach to Train Convolutional Neural Networks for Real-Time 6-DOF Camera Relocalization

Video Stream Relocalization With Deep Learning

Delving deeper into convolutional neural networks for camera relocalization.

From local understanding to global regression in monocular visual odometry

PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization

Indoor Relocalization in Challenging Environments with Dual-Stream Convolutional Neural Networks

Night-time indoor relocalization using depth image with Convolutional Neural Networks

A 6-Dofs Event-Based Camera Relocalization System by CNN-LSTM and Image Denoising

Accurate Rgb Camera Relocalization Using Regression Forest

Scene Coordinate Regression with Angle-Based Reprojection Loss for Camera Relocalization

Deep 6-DoF camera relocalization in variable and dynamic scenes by multitask learning

Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization

Camera Re-localization by Training Multi-dataset Simultaneously Via Convolutional Neural Network

Deep Convolutional Neural Network for 6-DOF Image Localization

VNLSTM-PoseNet: A Novel Deep ConvNet for Real-Time 6-DOF Camera Relocalization in Urban Streets.

Predicting World Coordinates of Pixels in RGB Images Using Convolutional Neural Network for Camera Relocalization

Image-similarity-based Convolutional Neural Network for Robot Visual Relocalization

Relative geometry-aware siamese neural network for 6DOF camera relocalization

Towards CNN Map Compression for camera relocalisation