ArCSEM: Artistic Colorization of SEM Images via Gaussian Splatting

Takuma Nishimura,Andreea Dogaru,Martin Oeggerli,Bernhard Egger

2024-10-25

Abstract:Scanning Electron Microscopes (SEMs) are widely renowned for their ability to analyze the surface structures of microscopic objects, offering the capability to capture highly detailed, yet only grayscale, images. To create more expressive and realistic illustrations, these images are typically manually colorized by an artist with the support of image editing software. This task becomes highly laborious when multiple images of a scanned object require colorization. We propose facilitating this process by using the underlying 3D structure of the microscopic scene to propagate the color information to all the captured images, from as little as one colorized view. We explore several scene representation techniques and achieve high-quality colorized novel view synthesis of a SEM scene. In contrast to prior work, there is no manual intervention or labelling involved in obtaining the 3D representation. This enables an artist to color a single or few views of a sequence and automatically retrieve a fully colored scene or video. Project page: <a class="link-external link-https" href="https://ronly2460.github.io/ArCSEM" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The problem this paper attempts to solve is: how to utilize the 3D structural information of Scanning Electron Microscope (SEM) images to automatically propagate color information from a small number of manually colored views, achieving high-quality artistic coloring and new view synthesis of SEM images.

Specifically, the paper proposes a method named ArCSEM to achieve this goal through the following steps:
1. **3D Scene Representation**: Using 2D Gaussian Splatting (2DGS) technology to construct an accurate 3D scene representation from multi-view grayscale SEM images.
2. **Illumination Variation Handling**: Applying image-specific Affine Color Transformation (ACT) to handle illumination variations under different views.
3. **Color Propagation**: Utilizing a limited number of manually colored images to propagate color information throughout the entire 3D scene via pseudo-color and semantic correspondences.
4. **Optimization and Refinement**: Further enhancing the quality of the coloring by fixing the geometric structure and optimizing the spherical harmonic coefficients.

This method not only reduces the workload of manual coloring but also generates high-quality new view images, allowing artists to achieve more creative expressions in the microscopic world.