Multilayer Perceptron-Based Fusion Method for Metal Surface Measurement Data by Multi-Sensors Incorporating Photometric Stereo and Structured Light
Yang Yuansong,Wang Xi,Ren Mingjun
DOI: https://doi.org/10.3788/AOS230497
2023-01-01
Acta Optica Sinica
Abstract:Objective Accurate reconstruction of the workpiece surface is crucial to evaluate the machining process and control product quality. Non-contact three-dimensional (3D) reconstruction methods are widely employed due to their low cost, high measurement speed, and high measurement accuracy. Common non-contact reconstruction methods, such as structured light technology, photogrammetry, and laser scanning, generally assume a diffuse surface. However, the machined metal surfaces obviously deviate from this assumption and exhibit complicated high light reflectance, which incurs heavy high-frequency noises and numerous invalid measurement data. Although high dynamic structured light research works focus on this challenge, all these methods fail to break the assumption of diffuse surface, and thus the measurement efficiency is low, and the recovery of missing measurement data is limited. Compared with these methods, photometric stereo can estimate the complete surface normal of the metal surface and overcome the influence of non-diffuse reflectance by inversely modeling the reflectance. However, the error accumulation happens during the normal integration, resulting in distorted shapes. In order to solve these problems, a multilayer perceptron-based fusion method is proposed for metal surface measurement by multi-sensors incorporating photometric stereo and structured light. The point cloud by structured light offers the geometry constraint, and the surface normal by photometric stereo provides the texture constraint. As a result, the accurate 3D reconstruction of the metal surface is achieved by fusing the point cloud and the surface normal. Methods In this paper, one multi-sensor system including several light-emitting diodes (LEDs), one projector, and one camera is designed, and the structured light measurement is achieved by the projector and the camera. The photometric stereo system consists of the camera and the LEDs. These two measurement sensors share the same camera, and thus the error of coordinate system matching is avoided. The structured light gives the imperfect noisy point cloud, and the complete surface normal with small high-frequency noises is estimated by the photometric stereo. In order to fuse these two different measurement data, a self-supervised multilayer perceptron network based on position encoding is designed according to the principle of normal integration, which achieves the mapping from the pixel coordinate to the depth under the camera coordinate. In the training process, the point cloud provides shape supervision, and the accurate surface normal gives the texture supervision to complete the point cloud. Thus, the complete and highly accurate 3D reconstructions of metal surfaces are output. Both synthetic experiments and real experiments verify the effectiveness of the proposed fusion method. Results and Discussions In this paper, synthetic experiments are designed to test the influence of different surface normal estimation errors on the proposed fusion method and verify the superiority of the proposed method to normal integration, especially under the condition of the noisy surface normal. The estimation error of structured light is synthesized by adding the Gaussian noise with a mean value of 0 and a standard deviation of 0. 040 mm. In order to simulate different levels of surface normal estimation errors, the Gaussian noise with mean values of 0. 01, 0. 03, 0. 05, and 0. 10 and standard deviation of 0. 025 is imposed respectively. With the increase in the surface normal estimation error, the proposed method can maintain excellent reconstruction accuracy, but the conventional normal integration algorithm exhibits dramatic degradation (Fig. 11). For different noise levels, the accuracy of the proposed fusion method enhances by 96. 6%, 96. 4%, 96. 2%, and 92. 4% compared with that of normal integration. In the real experiments, the point cloud and normal vector data are measured by the multi-sensors system incorporating the photometric stereo and structured light. As exhibited in Fig. 14, the reconstruction results of normal integration obviously distort due to the cumulative error, the point cloud by the structured light has missing data and high-frequency noises. The proposed fusion method effectively avoids the cumulative error of normal integration, completes the invalid data, and reduces the measurement error. According to the measurement results of the coordinate measurement machine (CMM), the accuracy of the proposed method enhances by about 50. 4% compared with that of the structured light measurement (Fig. 15). Conclusions In this paper, a multilayer perceptron-based fusion method is proposed for the measurement data of the multi-sensor system incorporating photometric stereo and structured light. In the metal surface measurement, the point cloud and surface normal are obtained by the multi-sensor, and a multilayer perceptron network based on position encoding is designed to achieve the final measurement. In order to effectively fuse the point cloud and the surface normal, the point cloud is employed as the shape constraint, and the surface normal vector is adopted as the texture constraint to supervise the proposed network. The synthetic experiments prove that the accuracy of the proposed fusion method improves by over 90% than that of the normal integration under the condition of noisy normal. The real experiments indicate that the proposed method can simultaneously filter out the high-frequency noise and complete the invalid measurement data. Besides, compared with the structured light reconstruction results, the accuracy of the proposed methods improves by about 50. 4% based on the measurement results of the CMM. Future research work can further analyze the uncertainty of the proposed multi-sensor systems.