Abstract:We present an approach to backpropagating through minimal problem solvers in end-to-end neural network training. Traditional methods relying on manually constructed formulas, finite differences, and autograd are laborious, approximate, and unstable for complex minimal problem solvers. We show that using the Implicit function theorem (IFT) to calculate derivatives to backpropagate through the solution of a minimal problem solver is simple, fast, and stable. We compare our approach to (i) using the standard autograd on minimal problem solvers and relate it to existing backpropagation formulas through SVD-based and Eig-based solvers and (ii) implementing the backprop with an existing PyTorch Deep Declarative Networks (DDN) framework. We demonstrate our technique on a toy example of training outlier-rejection weights for 3D point registration and on a real application of training an outlier-rejection and RANSAC sampling network in image matching. Our method provides $100\%$ stability and is 10 times faster compared to autograd, which is unstable and slow, and compared to DDN, which is stable but also slow.
What problem does this paper attempt to address?
This paper explores the problem of backpropagation with minimal problem solvers in end-to-end neural network training. Traditional manual formula construction, finite difference, and automatic differentiation methods are both tedious and inaccurate for complex minimal problem solvers and unstable. The authors propose using the Implicit Function Theorem (IFT) to calculate derivatives, thus achieving more stable and faster backpropagation through minimal problem solvers.
In the introduction, the author points out that recent minimal problem solvers have been incorporated into end-to-end machine learning pipelines, such as camera localization, image matching, and geometric model estimation. However, the differentiation between these solvers in neural network training is a challenge. Early methods include explicit derivative formulas and finite difference, while recent work adopts automatic differentiation. However, these methods may be unstable or slow for complex solver templates.
The main contribution of the paper is to demonstrate how to directly use IFT or PyTorch's Deep Declarative Networks (DDN) framework to implement backpropagation, improving stability and efficiency. Compared to standard automatic differentiation methods, their approach provides 100% stability in 3D point registration toy examples and real-world image matching applications, and is 10 times faster.
The paper also compares the backpropagation formulas using IFT with solvers based on SVD and Eig, as well as backpropagation using the DDN framework. Experimental results show that IFT backpropagation provides more stable results in complex templates and has advantages in computation speed.
In addition, the paper discusses the applications of IFT in end-to-end learning, such as camera relative pose estimation and training of RANSAC sampling networks, and compares them with existing methods, demonstrating the effectiveness and advantages of the proposed approach.