Efficient automated high dynamic range 3D measurement via deep reinforcement learning

Pan Zhang,Kai Zhong,Zhongwei Li,Yusheng Shi

DOI: https://doi.org/10.1364/oe.510515

IF: 3.8

2024-01-30

Optics Express

Abstract:Pan Zhang, Kai Zhong, Zhongwei Li, Yusheng Shi High dynamic range 3D measurement technology, utilizing multiple exposures, is pivotal in industrial metrology. However, selecting the ... [Opt. Express 32, 4857-4875 (2024)]

optics

What problem does this paper attempt to address?

Physics-based supervised learning method for high dynamic range 3D measurement with high fidelity

Fuqian Li,Xingman Niu,Jing Zhang,Qican Zhang,Yajun Wang

DOI: https://doi.org/10.1364/ol.506775

IF: 3.6

2024-01-26

Optics Letters

Abstract:Fuqian Li, Xingman Niu, Jing Zhang, Qican Zhang, Yajun Wang High dynamic range (HDR) 3D measurement is a meaningful but challenging problem. Recently, many deep-learning-based methods have been ... [Opt. Lett. 49, 602-605 (2024)]

optics
Deep learning based measurement accuracy improvement of high dynamic range objects in fringe projection profilometry

Mengmeng Shen,Liangbo He,Haihua Zhang,Lihong Ma,Yong Li

DOI: https://doi.org/10.1364/oe.538247

IF: 3.8

2024-09-19

Optics Express

Abstract:Mengmeng Shen, Liangbo He, Haihua Zhang, Lihong Ma, Yong Li One of the key factors affecting the accuracy of three-dimensional (3D) measurement in fringe projection profilometry (FPP) is the phase ... [Opt. Express 32, 35689-35702 (2024)]

optics
High-speed high dynamic range 3D shape measurement based on deep learning

Liang Zhang,Qian Chen,Chao Zuo,Shijie Feng

DOI: https://doi.org/10.1016/j.optlaseng.2020.106245

IF: 5.666

2020-11-01

Optics and Lasers in Engineering

Abstract:<p>For many three-dimensional (3D) measurement techniques based on fringe projection profilometry (FPP), measuring the objects with a large variation range of surface reflectivity is always a very tricky problem due to the limited dynamic range of camera. Many high dynamic range (HDR) 3D measurement methods are developed for static scenes, which are fragile for dynamic objects. In this paper, we address the problem of phase information loss in HDR scenes, in order to enable 3D reconstruction from saturated or dark images by deep learning. By using a specifically designed convolutional neural network (CNN), we can accurately extract phase information in both the low signal-to-noise ratio (SNR) and saturation situations after proper training. Experimental results demonstrate the success of our network in 3D reconstruction for both static and dynamic HDR objects. Our method can improve the dynamic range of three-step phase-shifting by a factor of 4.8 without any additional projected images or hardware adjustment during measurement. And the final 3D measurement speed of our method is about 13.89 Hz (off-line).</p>

optics
Automated Exposures Selection for High Dynamic Range Structured-Light 3-D Scanning

Wenyuan Chen,Xingjian Liu,Changhai Ru,Yu Sun

DOI: https://doi.org/10.1109/tie.2022.3201318

IF: 7.7

2023-02-22

IEEE Transactions on Industrial Electronics

Abstract:Structured light scanning has gained widespread applications in industrial metrology; however, measuring objects that have large surface-reflectivity variations is challenging. For this purpose, the high dynamic range (HDR) technique has become a standard method to fuse features under multiple exposures. Presently, in HDR measurement, manual or cumbersome exposure selection strategies are used. This article reports a new exposure selection approach. An image quality metric is designed to evaluate captured images and suppress regional overexposure, based on which a multiple-exposures selection strategy is developed. Unlike existing methods for single-optimal exposure selection, in our approach, pixel filtering is introduced to locate eligible pixels and candidate pixels in next exposures. Experimental results with industrial parts having large surface-reflectivity variations demonstrated that compared with HDR with 15 exposures (equal time intervals), our proposed approach achieved a similar surface coverage rates (97.4% versus 98.1%) and measurement accuracy (0.043 versus 0.041 mm), but only took 3–4 exposures averagely.
Single-shot 3D measurement of highly reflective objects with deep learning

Lingbao Kong,Mingzhu Wan

DOI: https://doi.org/10.1364/OE.487917

IF: 3.8

2023-04-21

Optics Express

Abstract:MingZhu Wan, Lingbao Kong Three-dimensional (3D) measurement methods based on fringe projection profilometry (FPP) have been widely applied in industrial ... [Opt. Express 31, 14965-14985 (2023)]

optics
Learning from General Diffuse Surfaces: An Event-driven Approach for High Dynamic Range Industrial Optical Measurement

Yuhui Li,Chen Xu,Lilin Liu

DOI: https://doi.org/10.1016/j.optlastec.2024.111183

IF: 4.939

2024-10-01

Optics & Laser Technology

Abstract:Employing the most concise physical systems to producing high-precision object properties—phase and amplitude—is essential for high dynamic range (HDR) optical measurements but presents significant challenges. The commonly used multi-exposure or multi-fringe pattern intensity adjustment techniques, based on frame cameras, often sacrifice the efficiency of the physical system to achieve computational gains. Existing frame-based deep learning HDR 3D optical measurement schemes have demonstrated their vast potential in executing this task, yet they typically struggle with the inefficiency of producing training datasets and the complexity of input selections for neural networks. This paper introduces a novel, deep learning-based event-driven high dynamic range sensing (eDL-HDRS) framework that utilizes a streamlined event-camera-based structured light system. Learning only from general diffuse surfaces, HDR scenes can be inferred effectively. With the developed spatio-temporal consistency event representation algorithm and advanced optimization strategies, the proposed eDL-HDRS seamlessly learns the intrinsic physical model of the structured light system, enabling a universal mapping from degraded event streams to advanced visual tasks such as amplitude imaging and 3D reconstruction.

optics,physics, applied
Efficient 3D measurement of a HDR surface based on adaptive fringe projection

Jialing Hu,Jiangping Zhu,Pei Zhou

DOI: https://doi.org/10.1364/AO.470064

IF: 1.9

2022-10-20

Applied Optics

Abstract:Jialing Hu, Jiangping Zhu, Pei Zhou 3D measurement methods of a high-dynamic-range (HDR) surface based on adaptive fringe projection have aroused extensive research interest. ... [Appl. Opt. 61, 9028-9036 (2022)]

optics
Automatic optimization of projection intensity for high dynamic range 3D surface measurement

Renhao Zheng,Lijing Li,Maosen Wan,Wei Zhang,Liandong Yu

DOI: https://doi.org/10.1016/j.optlaseng.2023.107888

IF: 5.666

2024-02-29

Optics and Lasers in Engineering

Abstract:Stripe projection profilers are widely used in many fields because of their high speed, high accuracy, and clear surface textures. In the structured light projection method, the complex reflectance distribution on the surface of the high-dynamic-range objective is an important factor that affects the measurement accuracy, as it can result in image over- and under-exposure. In this study, a method is proposed to adaptively determine the number of stripe patterns and the corresponding number of light intensities of the stripe patterns based on the complex reflectance of the surface of the object being measured. A multi threshold Otsu algorithm is used to classify the histogram of the surface reflectance distribution of the measured object, and several sets of stripe patterns with optimal light intensity are generated according to the light intensity response function. The original stripe patterns acquired at different light intensities are synthesized pixel by pixel into a high dynamic range (HDR) stripe image, and a four-step phase-shifting algorithm is used to obtain the unwrapped phase from the synthesized image. Experimental results show that this method can accurately measure the target of a surface-reflectance-transformed HDR.

optics
High dynamic range 3D measurements with fringe projection profilometry: a review

Shijie Feng,Liang Zhang,Chao Zuo,Tianyang Tao,Qian Chen,Guohua Gu

DOI: https://doi.org/10.1088/1361-6501/aae4fb

IF: 2.398

2018-11-05

Measurement Science and Technology

Abstract:In optical metrology, fringe projection is considered to be one of the most reliable techniques for recovering the shape of objects. For this technique, however, it is challenging to measure objects with a large variation in surface reflectivity, e.g. a scenario containing both dark and bright objects. Researchers have thus developed various approaches to high dynamic range (HDR) three-dimensional (3D) measurements over the years. In this paper, we present an overview of these techniques, as well as a new and definitive classification for them. We implement a set of representative techniques to measure objects with different characteristics of reflectance and discuss the advantages and constraints of the techniques according to the comparative results. Moreover, challenging problems and future research directions are discussed to advance HDR 3D measurement techniques.

engineering, multidisciplinary,instruments & instrumentation
Adaptive Light Estimation using Dynamic Filtering for Diverse Lighting Conditions

Junhong Zhao,Andrew Chalmers,Taehyun Rhee

DOI: https://doi.org/10.1109/tvcg.2021.3106497

IF: 5.2

2021-11-01

IEEE Transactions on Visualization and Computer Graphics

Abstract:High dynamic range (HDR) panoramic environment maps are widely used to illuminate virtual objects to blend with real-world scenes. However, in common applications for augmented and mixed-reality (AR/MR), capturing 360° surroundings to obtain an HDR environment map is often not possible using consumer-level devices. We present a novel light estimation method to predict 360° HDR environment maps from a single photograph with a limited field-of-view (FOV). We introduce the Dynamic Lighting network (DLNet), a convolutional neural network that dynamically generates the convolution filters based on the input photograph sample to adaptively learn the lighting cues within each photograph. We propose novel Spherical Multi-Scale Dynamic (SMD) convolutional modules to dynamically generate sample-specific kernels for decoding features in the spherical domain to predict 360° environment maps. Using DLNet and data augmentations with respect to FOV, an exposure multiplier, and color temperature, our model shows the capability of estimating lighting under diverse input variations. Compared with prior work that fixes the network filters once trained, our method maintains lighting consistency across different exposure multipliers and color temperature, and maintains robust light estimation accuracy as FOV increases. The surrounding lighting information estimated by our method ensures coherent illumination of 3D objects blended with the input photograph, enabling high fidelity augmented and mixed reality supporting a wide range of environmental lighting conditions and device sensors.

computer science, software engineering
High dynamic range 3D measurements based on space-time speckle correlation and color camera

Tong Li,Shaohui Zhang,Yao Hu,Qun Hao

DOI: https://doi.org/10.1364/OE.440105

2021-10-25

Abstract:Structured light (SL) based three-dimensional (3D) measurement struggles to estimate high dynamic range (HDR) scenes, where both high and low reflectivity parts exist simultaneously. This paper proposes a method through the joint design and optimization of hardware and algorithms, in which only four frames are required to realize the 3D reconstruction of HDR scenes. The height information of each sub-area in the scene under test can be encoded effectively by temporally projecting two sets of complementary speckle patterns onto target surface. To decode the corresponding patterns captured by the cameras, we design a stereo matching strategy consisting of space-time binary feature (ST-BIF) descriptor preliminary screening and zero-mean normalized cross-correlation (ST-ZNCC) final retrieval. The ST-BIF descriptor based on neighborhood comparison is designed to describe the space-time relative intensity change of projected speckles. Besides the HDR adaptability, the ST-BIF descriptor can effectively improve the matching speed. In addition, the measurable dynamic range can be further improved by fusing all channel disparities as evaluated results, benefitting from the different response of R, G and B channels in color camera to monochromatic light. Experiments are conducted to demonstrate the feasibility of the proposed method. The results indicate that our method achieves the root mean square error 0.2516mm (vs. 1.0668 by commonly used ZNCC) and an average coverage rate up to 94.87% (vs. 93.35% by commonly used ZNCC). Furthermore, the experimental results show that the proposed method can achieve 3D reconstruction of HDR scenes including specular reflection region.
Robust 3D Reconstruction Using HDR-Based SLAM

Chia-Hung Yeh,Min-Hui Lin

DOI: https://doi.org/10.1109/access.2021.3051257

IF: 3.9

2021-01-01

IEEE Access

Abstract:3D reconstruction is an important topic in the field of the emerging applications such as smart robotics, virtual reality (VR), augmented reality (AR), and autonomous driving. RGB-D simultaneous localization and mapping (SLAM) technique is widely used in the reconstruction process. However, low light and low textured environment often results in insufficient point features and fails the reconstruction. To address this problem, we propose a robust RGB-D SLAM system using high dynamic range (HDR) image information called HDR-based SLAM. The deep learning based HDR generation method is adopted to map a single low dynamic range (LDR) image into a radiance map which is normalized to exclude the influence of exposure time. We retrained the ORB descriptor patch to fit the normalized radiance maps in the feature matching step. The proposed method can improve the quantitative camera trajectory accuracy and qualitative result of geometry reconstruction. Experimental results show that the proposed method has better performance compared to that of the standard range imaging SLAM under challenging low light environment, which helps expand the applicability of 3D reconstruction system.

computer science, information systems,telecommunications,engineering, electrical & electronic
High-Dynamic-Range 3-D Shape Measurement With Adaptive Speckle Projection Through Segmentation-Based Mapping

Pei Zhou,Yue Cheng,Jiangping Zhu,Jialing Hu

DOI: https://doi.org/10.1109/tim.2022.3218110

IF: 5.6

2023-01-20

IEEE Transactions on Instrumentation and Measurement

Abstract:The 3-D shape measurement of high-dynamic-range (HDR) scenes based on structured light projection is challenging because the conventional projection without considering the reflection distribution of the tested surface usually causes over-exposure and over-dark areas simultaneously in the captured images, which will lead to the failure or low accuracy of 3-D reconstruction. In this article, we propose a novel HDR 3-D surface measurement method based on adaptive speckle projection with fully considering the reflection distribution. An efficient segmentation-based mapping strategy is established to speed up the generation of adaptive speckle patterns, which segments the estimated mask with optimal projection intensity into multiple subregions by the eight-connected standard to raise mapping accuracy and project gray-coded and sinusoidal fringe patterns to build the pixelwise mapping relationship between binocular cameras and the projector for each subregion, respectively. Then the pixelwise intensity of projected speckle patterns can be accurately adjusted according to optimal intensity. The generated adaptive speckle pattern is used to scan the tested object, and high signal-to-noise-ratio (SNR) speckle images can be obtained for accurate and complete 3-D measurement. Experimental results verify that the point cloud produced by our method has obvious advantages compared with the traditional methods in terms of measurement accuracy, whose coverage is as large as 92.26% and the average distance (AD) relative to ground truth (GT) is as low as 0.128 mm. Meanwhile, the number of projected patterns for mapping is decreased by a third compared with the widely used mapping strategy.

engineering, electrical & electronic,instruments & instrumentation
High-precision dynamic three-dimensional shape measurement of specular surfaces based on deep learning

Zhaoxing Wu,Jie Wang,Xuan Jiang,Luyao Fan,Chen Wei,Huimin Yue,Yong Liu

DOI: https://doi.org/10.1364/oe.486101

IF: 3.8

2023-05-11

Optics Express

Abstract:Zhaoxing Wu, Jie Wang, Xuan Jiang, Luyao Fan, Chen Wei, Huimin Yue, Yong Liu In order to solve the difficulty of traditional phase measuring deflectometry (PMD) in considering precision and speed, an orthogonal ... [Opt. Express 31, 17437-17449 (2023)]

optics
High-speed 3D shape measurement from noisy fringe images using deep learning

Jiaye J. Wang,Yuzhen Zhang

DOI: https://doi.org/10.1117/12.2606719

2021-11-24

Abstract:High-speed three-dimensional (3D) shape measurement has become a very important technology in industrial manufacturing, motion detection and other scientific research. Although there are some methods to measure 3D surface patterns, it is still difficult to accurately measure the rapidly changing 3D high-speed scenes. Multi-frequency phase unwrapping usually uses a combination of noisy fringe images with different fringe frequencies for phase unwrapping, which has high accuracy and reliability. Benefiting from the success of deep learning in the field of computer vision in recent years, we combine multi-frequency phase-shifting and phase unwrapping with deep learning, and propose the high-speed 3D shape measurement from noisy fringe images using deep learning. Compared with traditional methods, this method can achieve more convenient and robust phase retrieval at high speed. Based on a good training model, the deep learning neural network can directly achieve the corresponding high-quality phase results after extensive learning of the data set collected at high speed. The experimental results demonstrate that this method can achieve 3D shape of the measured object with an accuracy of about 51m at the camera frame rate of 700 frames per second.
Multi-view high-dynamic-range 3D reconstruction and point cloud quality evaluation based on dual-frame difference images

Yun Feng,Rongyu Wu,Peiwu Li,Wenlei Wu,Jiahao Lin,Xiaojun Liu,Liangzhou Chen

DOI: https://doi.org/10.1364/ao.533718

IF: 1.9

2024-10-15

Applied Optics

Abstract:Yun Feng, Rongyu Wu, Peiwu Li, Wenlei Wu, Jiahao Lin, Xiaojun Liu, Liangzhou Chen A high dynamic range can easily lead to image saturation, making it a challenge for structured light 3D reconstruction. The article ... [Appl. Opt. 63, 7865-7874 (2024)]

optics
An investigation of high dynamic range imaging of highly reflective surfaces based on deep learning 3D fringe projection

wei gao,Shixiang Wang

DOI: https://doi.org/10.1117/12.3009055

2023-12-29

Abstract:Three-dimensional reconstruction of highly reflective surfaces has always been an important and difficult problem in structured light technology. In this paper, a novel method by combining optimal exposure with the adjustment of the intensity of projection fringes by deep learning is proposed a single frame method is used to extract the overexposed region in the optimal exposure time series for composite projection. By using the statistical histogram of the number of overexposed pixels of the measured object under different projection conditions, and combining with the proportion of pixels in the image that are not exposed, the optimal exposure time series is calculated and the overexposed area is automatically extracted. The single-frame composite high dynamic image is obtained by adjusting the projection intensity through absolute coordinates, which improves the single-frame high dynamic range technique for highly reflective surfaces. Several simulations and experiments were conducted. The experimental results show that the proposed method can effectively improve the 3D reconstruction accuracy of the highly reflective surface, and the number of effective pixels in the point cloud can reach more than 98%.

Engineering,Physics,Computer Science
Real-time intelligent 3D holographic photography for real-world scenarios

Xianlin Song,Jiaqing Dong,Minghao Liu,Zehao Sun,Zibang Zhang,Jianghao Xiong,Zilong Li,Xuan Liu,Qiegen Liu

DOI: https://doi.org/10.1364/oe.529107

IF: 3.8

2024-06-22

Optics Express

Abstract:Xianlin Song, Jiaqing Dong, Minghao Liu, Zehao Sun, Zibang Zhang, Jianghao Xiong, Zilong Li, Xuan Liu, Qiegen Liu Three-dimensional (3D) display can provide more information than two-dimensional display, and real-time 3D reconstruction of the real-world ... [Opt. Express 32, 24540-24552 (2024)]

optics
Optimization RGB-D 3-D Reconstruction Algorithm Based on Dynamic SLAM

Zihao Pan,Junyi Hou,Lei Yu

DOI: https://doi.org/10.1109/tim.2023.3248116

IF: 5.6

2023-03-11

IEEE Transactions on Instrumentation and Measurement

Abstract:This article proposes a robust dynamic feature segmentation simultaneous localization and mapping (SLAM) 3-D reconstruction algorithm based on the dynamic segmentation framework of DynaSLAM in order to realize the SLAM system mapping in the dynamic environment of intelligent robots. The algorithm employs the method of feature point reextraction to solve the problem of an insufficient number of original feature points. In the preprocessing stage, the optimized epipolar geometry model is designed in this article, and the image is segmented jointly with Mask R-CNN. At the point cloud postprocessing stage, a kernel principal component analysis (PCA) is used to reduce noise, and then, this article designs a dynamic filtering method based on octree for further processing. This processing effectively eliminates additional dynamic outliers. The results of experiments on TUM and Bonn datasets show that this algorithm achieves more than 10% better attitude estimation accuracy than the DynaSLAM algorithm in a highly dynamic environment and has advantages over the state-of-the-art (SOTA) algorithms.

engineering, electrical & electronic,instruments & instrumentation

Efficient automated high dynamic range 3D measurement via deep reinforcement learning

Physics-based supervised learning method for high dynamic range 3D measurement with high fidelity

Deep learning based measurement accuracy improvement of high dynamic range objects in fringe projection profilometry

High-speed high dynamic range 3D shape measurement based on deep learning

Automated Exposures Selection for High Dynamic Range Structured-Light 3-D Scanning

Single-shot 3D measurement of highly reflective objects with deep learning

Learning from General Diffuse Surfaces: An Event-driven Approach for High Dynamic Range Industrial Optical Measurement

Efficient 3D measurement of a HDR surface based on adaptive fringe projection

Automatic optimization of projection intensity for high dynamic range 3D surface measurement

High dynamic range 3D measurements with fringe projection profilometry: a review

Adaptive Light Estimation using Dynamic Filtering for Diverse Lighting Conditions

High dynamic range 3D measurements based on space-time speckle correlation and color camera

Robust 3D Reconstruction Using HDR-Based SLAM

High-Dynamic-Range 3-D Shape Measurement With Adaptive Speckle Projection Through Segmentation-Based Mapping

High-precision dynamic three-dimensional shape measurement of specular surfaces based on deep learning

High-speed 3D shape measurement from noisy fringe images using deep learning

Multi-view high-dynamic-range 3D reconstruction and point cloud quality evaluation based on dual-frame difference images

An investigation of high dynamic range imaging of highly reflective surfaces based on deep learning 3D fringe projection

Real-time intelligent 3D holographic photography for real-world scenarios

Optimization RGB-D 3-D Reconstruction Algorithm Based on Dynamic SLAM