Computational 3D topographic microscopy from terabytes of data per sample

Kevin C. Zhou,Mark Harfouche,Maxwell Zheng,Joakim Jönsson,Kyung Chul Lee,Ron Appel,Paul Reamey,Thomas Doman,Veton Saliu,Gregor Horstmeyer,Roarke Horstmeyer

2023-06-05

Abstract:We present a large-scale computational 3D topographic microscope that enables 6-gigapixel profilometric 3D imaging at micron-scale resolution across $>$110 cm$^2$ areas over multi-millimeter axial ranges. Our computational microscope, termed STARCAM (Scanning Topographic All-in-focus Reconstruction with a Computational Array Microscope), features a parallelized, 54-camera architecture with 3-axis translation to capture, for each sample of interest, a multi-dimensional, 2.1-terabyte (TB) dataset, consisting of a total of 224,640 9.4-megapixel images. We developed a self-supervised neural network-based algorithm for 3D reconstruction and stitching that jointly estimates an all-in-focus photometric composite and 3D height map across the entire field of view, using multi-view stereo information and image sharpness as a focal metric. The memory-efficient, compressed differentiable representation offered by the neural network effectively enables joint participation of the entire multi-TB dataset during the reconstruction process. To demonstrate the broad utility of our new computational microscope, we applied STARCAM to a variety of decimeter-scale objects, with applications ranging from cultural heritage to industrial inspection.

Optics,Computer Vision and Pattern Recognition,Machine Learning,Image and Video Processing

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to develop a technology capable of 3D imaging large - area macroscopic surfaces with micron - level resolution. Specifically, existing 3D surface profilometry techniques usually can only cover a relatively small field of view (such as millimeter - level) at high resolution, and high - resolution imaging of a large field of view needs to be achieved through scanning and stitching, which is a relatively slow and complex process. Therefore, the paper proposes a new computational 3D topography microscope system - STARCAM (Scanning Topographic All - in - focus Reconstruction with a Computational Array Microscope), aiming to overcome these limitations and achieve fast, high - resolution large - area 3D imaging.

### Main Objectives:
1. **Improve the Resolution and Field of View of 3D Imaging**: Develop a technology capable of 3D imaging with micron - level resolution over a large area (more than 110 square centimeters).
2. **Reduce Data Acquisition Time**: Significantly reduce data acquisition time through a parallel multi - camera array design.
3. **Process Large - Scale Data Sets**: Develop a self - supervised neural network algorithm that can efficiently process multi - terabyte (TB) data sets to generate high - quality 3D height maps and all - in - focus images.

### Solutions:
- **Hardware Design**: Use a multi - camera array (MCAM) consisting of 54 cameras, each equipped with a high - resolution CMOS sensor and an appropriate lens configuration to achieve high - resolution imaging.
- **Data Acquisition**: Perform multi - dimensional scanning of samples through a 3 - axis translation stage to capture a 2.1 TB data set for each sample, containing 224,640 9.4 - megapixel images.
- **Computational Reconstruction**: Develop an algorithm based on a self - supervised neural network, using multi - view stereo vision and focus sharpness information for 3D reconstruction and stitching to generate a 6 - gigapixel (GP) all - in - focus image and a 3D height map.

### Application Prospects:
- **Cultural Heritage Protection**: It can be used for high - resolution 3D digitization of artworks, which is helpful for cultural relic protection and research.
- **Industrial Inspection**: It is suitable for high - resolution 3D inspection of electronic components, printed circuit boards (PCB), and chip - scale packages (CSP), etc., improving production quality control.
- **Other Applications**: It can also be applied in fields such as biomedicine and materials science to provide high - precision 3D surface information.

Through these innovations, the STARCAM system not only improves the resolution and field of view of 3D imaging but also greatly shortens the data acquisition and processing time, providing a powerful tool for applications in multiple fields.