Fusion of Multimodal Imaging and 3D Digitization Using Photogrammetry

Roland Ramm,Pedro de Dios Cruz,Stefan Heist,Peter Kühmstedt,Gunther Notni

DOI: https://doi.org/10.3390/s24072290

IF: 3.9

2024-04-04

Sensors

Abstract:Multimodal sensors capture and integrate diverse characteristics of a scene to maximize information gain. In optics, this may involve capturing intensity in specific spectra or polarization states to determine factors such as material properties or an individual's health conditions. Combining multimodal camera data with shape data from 3D sensors is a challenging issue. Multimodal cameras, e.g., hyperspectral cameras, or cameras outside the visible light spectrum, e.g., thermal cameras, lack strongly in terms of resolution and image quality compared with state-of-the-art photo cameras. In this article, a new method is demonstrated to superimpose multimodal image data onto a 3D model created by multi-view photogrammetry. While a high-resolution photo camera captures a set of images from varying view angles to reconstruct a detailed 3D model of the scene, low-resolution multimodal camera(s) simultaneously record the scene. All cameras are pre-calibrated and rigidly mounted on a rig, i.e., their imaging properties and relative positions are known. The method was realized in a laboratory setup consisting of a professional photo camera, a thermal camera, and a 12-channel multispectral camera. In our experiments, an accuracy better than one pixel was achieved for the data fusion using multimodal superimposition. Finally, application examples of multimodal 3D digitization are demonstrated, and further steps to system realization are discussed.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation

What problem does this paper attempt to address?

The paper aims to address the issue of fusing multimodal image data with 3D surface data. Specifically, the authors propose a new method to overlay image data captured by a low-resolution multimodal camera onto a high-resolution 3D model created through multi-view photogrammetry (MVP).

**The main contributions include:**
1. **Fusion Technology**: A method is proposed to achieve the fusion of multimodal images with 3D models without the need for feature matching. By pre-calibrating the geometric relationship, multimodal images are directly projected onto the 3D model.
2. **System Design**: An experimental setup is designed, which includes a high-resolution photo camera and a low-resolution multimodal camera, both fixed on a single mount. In this way, high-quality 3D models can be reconstructed, and multimodal information can be overlaid as a texture layer on the model.
3. **Application Scenarios**: Examples of multimodal 3D digitization applications are demonstrated, and steps for further system implementation are discussed.

Through this method, researchers can achieve multimodal 3D digitization of objects without relying on the resolution or image quality of the multimodal camera. This method is particularly suitable for situations that are difficult to handle with traditional techniques, such as very shiny or transparent objects.