Optical Metrology Embraces Deep Learning: Keeping an Open Mind

Bing Pan

DOI: https://doi.org/10.1038/s41377-022-00829-1

2022-01-01

Abstract:AbstractOptical metrology practitioners ought to embrace deep learning with an open mind, while devote continuing efforts to look for its theoretical groundwork and maintain an awareness of its limits.

What problem does this paper attempt to address?

Deep learning in optical metrology: a review

Chao Zuo,Jiaming Qian,Shijie Feng,Wei Yin,Yixuan Li,Pengfei Fan,Jing Han,Kemao Qian,Qian Chen

DOI: https://doi.org/10.1038/s41377-022-00714-x

2022-02-23

Light: Science and Applications

Abstract:Abstract With the advances in scientific foundations and technological implementations, optical metrology has become versatile problem-solving backbones in manufacturing, fundamental research, and engineering applications, such as quality control, nondestructive testing, experimental mechanics, and biomedicine. In recent years, deep learning, a subfield of machine learning, is emerging as a powerful tool to address problems by learning from data, largely driven by the availability of massive datasets, enhanced computational power, fast data storage, and novel training algorithms for the deep neural network. It is currently promoting increased interests and gaining extensive attention for its utilization in the field of optical metrology. Unlike the traditional “physics-based” approach, deep-learning-enabled optical metrology is a kind of “data-driven” approach, which has already provided numerous alternative solutions to many challenging problems in this field with better performances. In this review, we present an overview of the current status and the latest progress of deep-learning technologies in the field of optical metrology. We first briefly introduce both traditional image-processing algorithms in optical metrology and the basic concepts of deep learning, followed by a comprehensive review of its applications in various optical metrology tasks, such as fringe denoising, phase retrieval, phase unwrapping, subset correlation, and error compensation. The open challenges faced by the current deep-learning approach in optical metrology are then discussed. Finally, the directions for future research are outlined.
Investigating deep optics model representation in affecting resolved all-in-focus image quality and depth estimation fidelity

Xin Liu,Linpei LI,Xu Liu,Xiang Hao,Yifan Peng

DOI: https://doi.org/10.1364/OE.473084

IF: 3.8

2022-01-01

Optics Express

Abstract:The end-to-end (E2E) optimization of optics and image processing, dubbed deep optics, has renewed the state-of-the-art in various computer vision tasks. However, specifying the proper model representation or parameterization of the optical elements remains elusive. This article comprehensibly investigates three modeling hypotheses of the phase coded-aperture imaging under a representative context of deep optics, joint all-in-focus (AiF) imaging and monocular depth estimation (MDE). Specifically, we analyze the respective trade-off of these models and provide insights into relevant domain-specific requirements, explore the connection between the spatial feature of the point spread function (PSF) and the performance trade-off between the AiF and MDE tasks, and discuss the model sensitivity to possible fabrication errors. This study provides new prospects for future deep optics designs, particularly those aiming for AiF and/or MDE.(c) 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
4-K-resolution Minimalist Optical System Design Based on Deep Learning.

Dexiao Meng,Yan Zhou,Jian Bai

DOI: https://doi.org/10.1364/ao.510860

IF: 1.9

2023-01-01

Applied Optics

Abstract:In order to simplify optical systems, we propose a high-resolution minimalist optical design method based on deep learning. Unlike most imaging system design work, we combine optical design more closely with image processing algorithms. For optical design, we separately study the impact of different aberrations on computational imaging and then innovatively propose an aberration metric and a spatially micro-variant design method that better meet the needs of image recognition. For image processing, we construct a dataset based on the point spread function (PSF) imaging simulation method. In addition, we use a non-blind deblurring computational imaging method to repair spatially variant aberrations. Finally, we achieve clear imaging at 4 K (5184×3888) using only two spherical lenses and achieve image quality similar to that of complex lenses on the market.
Deep Learning Algorithm and Its Application in Optics

Zhou Hongqiang,Huang Lingling,Wang Yongtian

DOI: https://doi.org/10.3788/irla201948.1226004

2019-01-01

Abstract:As an important branch of machine learning, deep learning has reached another climax of machine learning since its inception. Deep learning has excellent performance in many fields such as image recognition and classification, semantic segmentation, and intelligent driving and so on. At the same time, deep learning algorithms are widely used in the field of optics such as computational hologram generation and imaging, non-parameter reconstruction of digital holography, and spectral resonance curves prediction due to their abstract feature recognition and extraction characteristics, strong model building and generalization capabilities. This article detailed the basic principles of deep learning and its typical application research in image classification, super-resolution imaging, computer generated hologram and digital holography, prediction of surface plasmonics resonance curves, and structural design of metasurfaces. And future development of deep learning in the physical optical field was worth exploring.
Applying machine learning to optical metrology: a review

Ruidong Xue,Helia Hooshmand,Mohammed Isa,Samanta Piano,Richard K Leach

DOI: https://doi.org/10.1088/1361-6501/ad7878

IF: 2.398

2024-09-11

Measurement Science and Technology

Abstract:This literature review investigates the integration of machine learning (ML) into optical metrology, unveiling enhancements in both efficiency and effectiveness of measurement processes. With a focus on phase demodulation, unwrapping, and phase-to-height conversion, the review highlights how ML algorithms have transformed traditional optical metrology techniques, offering improved speed, accuracy, and data processing capabilities. Efficiency improvements are underscored by advancements in data generation, intelligent sampling, and processing strategies, where ML algorithms have accelerated the metrological evaluations. Effectiveness is enhanced in measurement precision, with ML providing robust solutions to complex pattern recognition and noise reduction challenges. Additionally, the role of parallel computing using graphics processing units (GPUs) and field programmable gate arrays (FPGAs) is emphasised, showcasing their importance in supporting the computationally intensive ML algorithms for real-time processing. This review culminates in identifying future research directions, emphasising the potential of advanced ML models and broader applications within optical metrology. Through this investigation, the review articulates a future where optical metrology, empowered by ML, achieves improved levels of operational efficiency and effectiveness.

engineering, multidisciplinary,instruments & instrumentation
Toward a Thinking Microscope: Deep Learning in Optical Microscopy and Image Reconstruction

Yair Rivenson,Aydogan Ozcan

DOI: https://doi.org/10.48550/arXiv.1805.08970

IF: 5.414

2018-05-23

Machine Learning

Abstract:We discuss recently emerging applications of the state-of-art deep learning methods on optical microscopy and microscopic image reconstruction, which enable new transformations among different modes and modalities of microscopic imaging, driven entirely by image data. We believe that deep learning will fundamentally change both the hardware and image reconstruction methods used in optical microscopy in a holistic manner.
Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy

Kevin de Haan,Y. Rivenson,Yichen Wu,A. Ozcan

DOI: https://doi.org/10.1109/JPROC.2019.2949575

IF: 20.6

2020-01-01

Proceedings of the IEEE

Abstract:In recent years, deep learning has been shown to be one of the leading machine learning techniques for a wide variety of inference tasks. In addition to its mainstream applications, such as classification, it has created transformative opportunities for image reconstruction and enhancement in optical microscopy. Some of these emerging applications of deep learning range from image transformations between microscopic imaging systems to adding new capabilities to existing imaging techniques, as well as solving various inverse problems based on microscopy image data. Deep learning is helping us move toward data-driven instrument designs that blend microscopy and computing to achieve what neither can do alone. This article provides an overview of some of the recent work using deep neural networks to advance computational microscopy and sensing systems, also covering their current and future biomedical applications.

Physics,Engineering,Computer Science
Deep Neural Networks for Computational Optical Form Measurements

Lara Hoffmann,Clemens Elster

DOI: https://doi.org/10.5194/jsss-9-301-2020

2020-07-01

Abstract:Deep neural networks have been successfully applied in many different fields like computational imaging, medical healthcare, signal processing, or autonomous driving. In a proof-of-principle study, we demonstrate that computational optical form measurement can also benefit from deep learning. A data-driven machine learning approach is explored to solve an inverse problem in the accurate measurement of optical surfaces. The approach is developed and tested using virtual measurements with known ground truth.

Image and Video Processing,Machine Learning,Instrumentation and Detectors
Deep Learning for Photonic Design and Analysis: Principles and Applications

Bing Duan,Bei Wu,Jin-hui Chen,Huanyang Chen,Da-Quan Yang

DOI: https://doi.org/10.3389/fmats.2021.791296

2022-01-06

Frontiers in Materials

Abstract:Innovative techniques play important roles in photonic structure design and complex optical data analysis. As a branch of machine learning, deep learning can automatically reveal the inherent connections behind the data by using hierarchically structured layers, which has found broad applications in photonics. In this paper, we review the recent advances of deep learning for the photonic structure design and optical data analysis, which is based on the two major learning paradigms of supervised learning and unsupervised learning. In addition, the optical neural networks with high parallelism and low energy consuming are also highlighted as novel computing architectures. The challenges and perspectives of this flourishing research field are discussed.

materials science, multidisciplinary
Deep learning in optics and photonics for statistical inference, computing, and inverse design

Deniz Mengu,Md Sadman Sakib Rahman,Yi Luo,Onur Kulce,Aydogan Ozcan,jingxi li

DOI: https://doi.org/10.1364/aop.450345

IF: 24.75

2022-04-07

Advances in Optics and Photonics

Abstract:Deep learning has been revolutionizing information processing in many fields of science and engineering owing to the massively growing amounts of data and the advances in deep neural network architectures. As these neural networks are expanding their capabilities toward achieving state-of-the-art solutions for demanding statistical inference tasks in various applications, there appears to be a global need for low-power, scalable, and fast computing hardware beyond what existing electronic systems can offer. Optical computing might potentially address some of these needs with its inherent parallelism, power efficiency, and high speed. Recent advances in optical materials, fabrication, and optimization techniques have significantly enriched the design capabilities in optics and photonics, leading to various successful demonstrations of guided-wave and free-space computing hardware for accelerating machine learning tasks using light. In addition to statistical inference and computing, deep learning has also fundamentally affected the field of inverse optical/photonic design. The approximation power of deep neural networks has been utilized to develop optics/photonics systems with unique capabilities, all the way from nanoantenna design to end-to-end optimization of computational imaging and sensing systems. In this review, we attempt to provide a broad overview of the current state of this emerging symbiotic relationship between deep learning and optics/photonics.

optics
Deep Learning Based Computational Imaging: Status, Challenges, and Future

Zuo Chao,Feng Shijie,Zhang Xiangyu,Han Jing,Qian Chen

DOI: https://doi.org/10.3788/aos202040.0111003

2020-01-01

Acta Optica Sinica

Abstract:In recent years, optical imaging techniques have entered into the era of computational optical imaging from the traditional intensity and color imaging. Computational optical imaging, which is based on geometric optics, wave optics, and other theoretical foundations, establishes an accurate forward mathematical model for the whole image formation process of the scene imaged through the optical system and then sampled by the digital detector. Then, the high-quality reconstruction of the image and other high dimensional information, such as phase, spectrum, polarization, light field, coherence, refractive index, and three-dimension profile, which cannot be directly accessed using traditional methods, can be obtained through computational reconstruction method. However, the actual imaging performance of the computational imaging system is also limited by the "accuracy of the forward mathematical model" and "the reliability of inverse reconstruction algorithm ". Besides, the unpredictability of real physical imaging process and the complexity of solving high dimensional ill-posed inverse problems have become the bottleneck of further development of this field. In recent years, the rapid development of artificial intelligence and deep learning for the technology opens a new door for computational optical imaging technology. Unlike " physical driven" model that traditional computational imaging method is based on, computational imaging based on deep learning is a kind of "data-driven" method, which not only solves many problems considered quite challenge to be solved in this field, but also achieves remarkable improvement in information acquisition ability, imaging functions, and key performance indexes of imaging system, such as spatial resolution, temporal resolution, and detection sensitivity. This review first briefly introduces the current status and the latest progress of deep learning technology in the field of computational optical imaging. Then, the main problems and challenges faced by the current deep learning method in computational optical imaging field arc discussed. Finally, the future developments and possible research directions of this field arc prospected.
Deep learning for fast segmentation and critical dimension metrology & characterization enabling AR/VR design and fabrication

Kundan Chaudhary,Subhei Shaar,Raja Muthinti

2024-09-21

Abstract:Quantitative analysis of microscopy images is essential in the design and fabrication of components used in augmented reality/virtual reality (AR/VR) modules. However, segmenting regions of interest (ROIs) from these complex images and extracting critical dimensions (CDs) requires novel techniques, such as deep learning models which are key for actionable decisions on process, material and device optimization. In this study, we report on the fine-tuning of a pre-trained Segment Anything Model (SAM) using a diverse dataset of electron microscopy images. We employed methods such as low-rank adaptation (LoRA) to reduce training time and enhance the accuracy of ROI extraction. The model's ability to generalize to unseen images facilitates zero-shot learning and supports a CD extraction model that precisely extracts CDs from the segmented ROIs. We demonstrate the accurate extraction of binary images from cross-sectional images of surface relief gratings (SRGs) and Fresnel lenses in both single and multiclass modes. Furthermore, these binary images are used to identify transition points, aiding in the extraction of relevant CDs. The combined use of the fine-tuned segmentation model and the CD extraction model offers substantial advantages to various industrial applications by enhancing analytical capabilities, time to data and insights, and optimizing manufacturing processes.

Computer Vision and Pattern Recognition,Machine Learning,Image and Video Processing
How Optical CMMs and 3D Scanning Will Revolutionize the 3D Metrology World

Jean-Francois Larue,Daniel Brown,Marc Viala

DOI: https://doi.org/10.1007/978-1-4471-6741-9_5

2015-01-01

Abstract:Every day, optical technology plays a bigger and bigger role in metrology. No doubt optical metrology will soon be used to perform continuous, high-speed inspections of items of all shapes, sizes, and materials. However, it will take time to move from a world dominated by coordinate measuring machines that use sensors to one where optical CMM reigns, especially in industries such as automotive or aeronautics where every new thing must prove itself first. One of the biggest changes in metrology in the past 30 years has been the development of portable measuring devices. This has brought inspection right into the production line, as close to the part as possible. The change—sparked by the development of portable measuring arms in the early 1990s and the emergence of laser trackers shortly after—turned traditional industries’ inspection methods completely upside down. However, most portable measurement solutions still use technologies that have serious limitations in production environments. For example, these solutions demand extreme stability throughout the measurement process. Optical solutions, especially those using video cameras, sidestep these limitations by enabling automatic positioning and continuous measurement. Now, laser- and white-light-based digitizing technologies can also produce increasingly dense data. In this chapter, we will see the basic implementation principles and advantages offered by these optical solutions for performing direct inspections in a production environment. Key point is about using optical reflectors as metrological references on parts and using these references to (1) allow self-positioning of handheld scanners, (2) automatically align the device with a predetermined 3D reference, (3) easily move the device around the part, and (4) maintain high measurement precision in industrial production environments. We will also analyze all the factors that go into measurement precision and see how the use of optical technologies make it possible to greatly reduce the primary causes of measurement imprecision. To illustrate our analysis, the reader will find specific cases taken from real applications in the aeronautical, automotive, and naval industries.
Review of wavefront sensing technology in adaptive optics based on deep learning

Gao Zeyu,Jia Qiwang,Li Xinyang,Liang Ziqiang

DOI: https://doi.org/10.11884/HPLPB202133.210158

2021-08-15

Abstract:Wavefront sensing is an important part of adaptive optics system, which plays a key role in the fields of ground-based telescopes, laser transmission in atmosphere, wireless optical communication, laser nuclear fusion, and freeform surface optical measurement etc. Meanwhile, as a general advanced technology, deep learning has made revolutionary progress in many fields such as computer vision, natural language processing and so on. Using deep learning method to improve the wavefront sensor in adaptive optics system to achieve more accurate wavefront detection and adapt to more complex application scenarios is the development trend of adaptive optics, and also a new topic in the field of deep learning. This paper, introduces the application status of deep learning in adaptive optics wavefront sensing in detail. It also analyzes the research characteristics of different types of wavefront sensors, such as phase retrieval wavefront sensor and Shack-Hartmann wavefront sensor, and makes a summary at the end.

Engineering,Computer Science,Physics
Precision Optical Metrology and Smart Sensing

Xiangchao Zhang,Jian Wang,Yajun Wang

DOI: https://doi.org/10.3390/s24216816

IF: 3.9

2024-10-24

Sensors

Abstract:Optics, renowned for its non-contact measurement capabilities, versatility, and high sensitivity, is recognized as a powerful tool driving the technological evolution of precision metrology [...]

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Generating Starting Points for Designing Freeform Imaging Optical Systems Based on Deep Learning.

Wenchen Chen,Tong Yang,Dewen Cheng,Yongtian Wang

DOI: https://doi.org/10.1364/oe.432745

IF: 3.8

2021-01-01

Optics Express

Abstract:Deep learning is an important aspect of artificial intelligence and has been applied successfully in many optics-related fields. This paper proposes a generalized framework for generation of starting points for freeform imaging optical design based on deep learning. Compared with our previous work, this framework can be used for highly nonrotationally symmetric freeform refractive, reflective, and catadioptric systems. The system parameters can be advanced and the ranges of these system parameters can be wide. Using a special system evolution method and a K-nearest neighbor method, a full dataset consisting of the primary and secondary parts can be generated automatically. The deep neural network can then be trained in a supervised manner and can be used to generate good starting points directly. The convenience and feasibility of the proposed framework are demonstrated by designing a freeform off-axis three-mirror imaging system, a freeform off-axis four-mirror afocal telescope, and a freeform prism for an augmented reality near-eye display. The design framework reduces the designer's time and effort significantly and their dependence on advanced design skills. The framework can also be integrated into optical design software and cloud servers for the convenience of more designers.
Contrasting conventional and machine learning approaches to optical critical dimension measurements

Bryan M. Barnes,Mark-Alexander Henn

DOI: https://doi.org/10.1117/12.2551504

2020-03-24

Abstract:Accurate, optics-based measurement of feature sizes at deep sub-wavelength dimensions has been conventionally challenged by improved manufacturing, including smaller linewidths, denser layouts, and greater materials complexity at near-atomic scales. Electromagnetic modeling is relied upon heavily for forward maps used to solve the inverse problem of optical measurements for parametric estimation. Machine learning (ML) approaches are continually under consideration, either as a means to bypass direct comparison to simulation or as a method to augment nonlinear regression. In this work, ML approaches are investigated using a well-characterized experimental data set and its simulation library that assumes a 2-D geometry. The benefits and limitations of ML for optical critical dimension (OCD) metrology are illustrated by comparing a straightforward library lookup method and two ML approaches, a data-driven surrogate model for nonlinear regression using radial basis functions (RBF) and multiple-output Gaussian process regression (GPR) that indirectly applies the simulated intensity data. Both RBF and GPR generally improve accuracy over the conventional method with as few as 32 training points. However, as measurement noise is decreased the uncertainties from RBF and GPR differ greatly as the GPR posterior estimate of the variance appears to overestimate parametric uncertainties. Both accuracy and uncertainty must be addressed in OCD while balancing simulation versus ML computational requirements.
Deep-Learning-Aided Extraction of Optical Constants in Scanning Near-Field Optical Microscopy

Yueqi Zhao,Xinzhong Chen,Ziheng Yao,Mengkun Liu,Michael M. Fogler

DOI: https://doi.org/10.1063/5.0139517

2022-02-07

Abstract:Scanning near-field optical microscopy is one of the most effective techniques for spectroscopy of nanoscale systems. However, inferring optical constants from the measured near-field signal can be challenging because of a complicated and highly nonlinear interaction between the scanned probe and the sample. Conventional fitting methods applied to this problem often suffer from the lack of convergence or require human intervention. Here we develop an alternative approach where the optical parameter extraction is automated by a deep learning network. Compared to its traditional counterparts, our method demonstrates superior accuracy, stability against noise, and computational speed when applied to simulated near-field spectra.

Optics
An Insight on Optical Metrology in Manufacturing

Yuki Shimizu,Liang Chia Chen,Dae Wook Kim,Xiuguo Chen,Xinghui Li,Hiraku Matsukuma

DOI: https://doi.org/10.1088/1361-6501/abc578

2020-01-01

Abstract:Optical metrology is one of the key technologies in today’s manufacturing industry. In this article, we provide an insight into optical measurement technologies for precision positioning and quality assessment in today’s manufacturing industry. First, some optical measurement technologies for precision positioning are explained, mainly focusing on those with a multi-axis positioning system composed of linear slides, often employed in machine tools or measuring instruments. Some optical measurement technologies for the quality assessment of products are then reviewed, focusing on technologies for form measurement of products with a large metric structure, from a telescope mirror to a nanometric structure such as a semiconductor electrode. Furthermore, we also review the state-of-the-art optical technique that has attracted attention in recent years, optical coherence tomography for the non-destructive inspection of the internal structures of a fabricated component, as well as super-resolution techniques for improving the lateral resolution of optical imaging beyond the diffraction limit of light. This review article provides insights into current and future technologies for optical measurement in the manufacturing industry, which are expected to become even more important to meet the industry’s continuing requirements for high-precision and high-efficiency machining.
At the intersection of optics and deep learning: statistical inference, computing, and inverse design

Deniz Mengu,Md Sadman Sakib Rahman,Yi Luo,Jingxi Li,Onur Kulce,Aydogan Ozcan

DOI: https://doi.org/10.1364/AOP.450345

IF: 24.75

2022-05-20

Advances in Optics and Photonics

Abstract:Deep learning has been revolutionizing information processing in many fields of science and engineering owing to the massively growing amounts of data and the advances in deep neural network architectures. As these neural networks are expanding their capabilities toward achieving state-of-the-art solutions for demanding statistical inference tasks in various applications, there appears to be a global need for low-power, scalable, and fast computing hardware beyond what existing electronic systems can offer. Optical computing might potentially address some of these needs with its inherent parallelism, power efficiency, and high speed. Recent advances in optical materials, fabrication, and optimization techniques have significantly enriched the design capabilities in optics and photonics, leading to various successful demonstrations of guided-wave and free-space computing hardware for accelerating machine learning tasks using light. In addition to statistical inference and computing, deep learning has also fundamentally affected the field of inverse optical/photonic design. The approximation power of deep neural networks has been utilized to develop optics/photonics systems with unique capabilities, all the way from nanoantenna design to end-to-end optimization of computational imaging and sensing systems. In this review, we attempt to provide a broad overview of the current state of this emerging symbiotic relationship between deep learning and optics/photonics. © 2022 Optica Publishing Group

optics

Optical Metrology Embraces Deep Learning: Keeping an Open Mind

Deep learning in optical metrology: a review

Investigating deep optics model representation in affecting resolved all-in-focus image quality and depth estimation fidelity

4-K-resolution Minimalist Optical System Design Based on Deep Learning.

Deep Learning Algorithm and Its Application in Optics

Applying machine learning to optical metrology: a review

Toward a Thinking Microscope: Deep Learning in Optical Microscopy and Image Reconstruction

Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy

Deep Neural Networks for Computational Optical Form Measurements

Deep Learning for Photonic Design and Analysis: Principles and Applications

Deep learning in optics and photonics for statistical inference, computing, and inverse design

Deep Learning Based Computational Imaging: Status, Challenges, and Future

Deep learning for fast segmentation and critical dimension metrology & characterization enabling AR/VR design and fabrication

How Optical CMMs and 3D Scanning Will Revolutionize the 3D Metrology World

Review of wavefront sensing technology in adaptive optics based on deep learning

Precision Optical Metrology and Smart Sensing

Generating Starting Points for Designing Freeform Imaging Optical Systems Based on Deep Learning.

Contrasting conventional and machine learning approaches to optical critical dimension measurements

Deep-Learning-Aided Extraction of Optical Constants in Scanning Near-Field Optical Microscopy

An Insight on Optical Metrology in Manufacturing

At the intersection of optics and deep learning: statistical inference, computing, and inverse design