Abstract:Multi-perspective cameras with potentially non-overlapping fields of view have become an important exteroceptive sensing modality in a number of applications such as intelligent vehicles, drones, and mixed reality headsets. In this work, we challenge one of the basic assumptions made in these scenarios, which is that the multi-camera rig is rigid. More specifically, we are considering the problem of estimating the relative pose between a static non-rigid rig in different spatial orientations while taking into account the effect of gravity onto the system. The deformable physical connections between each camera and the body center are approximated by a simple cantilever model, and inserted into the generalized epipolar constraint. Our results lead us to the important insight that the latent parameters of the deformation model, meaning the gravity vector in both views, become observable. We present a concise analysis of the observability of all variables based on noise, outliers, and rig rigidity for two different algorithms. The first one is a vision-only alternative, while the second one makes use of additional gravity measurements. To conclude, we demonstrate the ability to sense gravity in a real-world example, and discuss practical implications.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to estimate the relative poses in different spatial attitudes in a non - rigid multi - view camera system, while taking into account the influence of gravity on the system. Specifically, the paper challenges the rigid structure usually assumed in multi - camera systems and proposes a new method to estimate the relative poses of a static non - rigid camera device in different spatial directions, while considering the deformation influence of gravity on the system. ### Main problems 1. **Relative pose estimation of non - rigid multi - view cameras**: - Traditional methods assume that the multi - view camera system is rigid, that is, the relative positions and orientations between cameras will not change. However, in practical applications, such as light - duty drones or fixed - wing drones, the camera device may be deformed due to the influence of external forces (especially gravity). - The paper proposes a new method to estimate the relative poses of such non - rigid multi - view camera systems by introducing a physical deformation model. 2. **Observability of the gravity vector**: - By introducing a physical deformation model, the paper proves that even without directly measuring gravity or the corresponding points inside the camera, the deformation model can be constrained by the observed image data and the direction of gravity can be estimated. - This finding means that the inertial state can be indirectly measured by geometric computer vision methods without additional environmental prior knowledge or an inertial measurement unit (IMU). ### Solutions 1. **Physical deformation model**: - The paper uses a simplified cantilever beam model to approximately describe the elastic connection between each camera and the central body. This model takes into account the influence of gravity on the camera position and orientation. - By introducing these deformation parameters into the generalized epipolar constraint, a new generalized relative pose estimation problem can be formed. 2. **Algorithm design**: - Two new algorithms are proposed to jointly estimate the generalized relative pose and the direction of gravity: - **Pure vision method**: Only image data is used for optimization, which is suitable for situations without additional sensors. - **IMU - combined method**: Additional IMU data is used to improve the estimation accuracy. 3. **Experimental verification**: - The effectiveness of the proposed model and algorithms is verified through simulation experiments. The experimental results show that this method can accurately estimate the relative pose and the direction of gravity even in the presence of noise and outliers. ### Practical applications - This method has important practical value in application scenarios such as light - duty drones, fixed - wing drones, and self - driving cars, especially in cases where high - precision relative pose estimation is required and no additional sensors can be used. ### Summary By introducing a physical deformation model, the paper successfully solves the relative pose estimation problem of non - rigid multi - view camera systems and proves the observability of the gravity vector. This achievement not only improves the accuracy of relative pose estimation but also provides new ideas for future geometric computer vision research.

Relative Pose for Nonrigid Multi-Perspective Cameras: The Static Case