Abstract:Diffusion models have found phenomenal success as expressive priors for solving inverse problems, but their extension beyond natural images to more structured scientific domains remains limited. Motivated by applications in materials science, we aim to reduce the number of measurements required from an expensive imaging modality of interest, by leveraging side information from an auxiliary modality that is much cheaper to obtain. To deal with the non-differentiable and black-box nature of the forward model, we propose a framework to train a multimodal diffusion model over the joint modalities, turning inverse problems with black-box forward models into simple linear inpainting problems. Numerically, we demonstrate the feasibility of training diffusion models over materials imagery data, and show that our approach achieves superior image reconstruction by leveraging the available side information, requiring significantly less amount of data from the expensive microscopy modality.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is in fields such as materials science, how to use the side information provided by auxiliary modalities (such as cheap and easily accessible polarized light imaging) to accelerate the main imaging process (such as expensive and time - consuming electron back - scatter diffraction imaging). Specifically, the authors propose a method to reduce the amount of data required from expensive imaging modalities while maintaining or improving the quality of image reconstruction by training a multimodal diffusion model. The core challenge of the paper is that, usually, the forward model (i.e., the function that converts the main imaging modality into the auxiliary modality) is a non - differentiable black - box model, which makes it difficult for existing diffusion - model - based inverse problem - solving methods to be directly applied. To solve this problem, the authors propose a framework that can transform this nonlinear inverse problem into a simple linear inpainting problem, so that image reconstruction can be carried out using the multimodal diffusion model without accessing the black - box forward model. ### Main contributions: 1. **Problem reformulation**: The authors reformulate the nonlinear inverse problem with a black - box forward model as a linear inpainting problem, and capture the joint distribution of different modalities by training a multimodal diffusion model. 2. **Experimental verification**: In materials science applications, the authors use polarized light imaging as side information to verify the superior performance of this method in reconstructing electron back - scatter diffraction images. Compared with models using only single - modal data, it significantly reduces the amount of data required for the main imaging modality. 3. **Extended application**: This work extends the application range of diffusion models and shows the potential of generative models in processing highly - structured scientific data. ### Mathematical formulas: - **Forward model**: \[ y = f(x^\star)+\xi \] where \(y\) is the measured value, \(f\) is the forward model, \(x^\star\) is the target image, and \(\xi\) is the measurement noise. - **Joint modality definition**: \[ X = [x\quad f(x)],\quad x\sim p^\star \] - **Linear inpainting model**: \[ Y = [y_{\text{main}}\quad y_{\text{aux}}]=\bar{P}_\Omega(X)+[0\quad \xi] \] ### Conclusion: By using the side information provided by the auxiliary modality, this method can significantly reduce the amount of data required from the expensive imaging modality while maintaining or improving the quality of image reconstruction. This not only has important application value in materials science, but also provides new ideas for multimodal data fusion in other fields.

Leveraging Multimodal Diffusion Models to Accelerate Imaging with Side Information

A Variational Perspective on Solving Inverse Problems with Diffusion Models

MRI Reconstruction with Side Information using Diffusion Models

Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

Multimodal Image Fusion Based on Diffusion Model

Enhancing and Accelerating Diffusion-Based Inverse Problem Solving through Measurements Optimization

Conditional score-based diffusion models for solving inverse problems in mechanics

General Intelligent Imaging and Uncertainty Quantification by Deterministic Diffusion Model

CT Reconstruction using Diffusion Posterior Sampling conditioned on a Nonlinear Measurement Model

Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Models

Diffusion model with disentangled modulations for sharpening multispectral and hyperspectral images

Learning Diffusion Model from Noisy Measurement using Principled Expectation-Maximization Method

Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements

Diff-IF: Multi-modality image fusion via diffusion model with fusion knowledge prior

Diffuse3D: Wide-Angle 3D Photography Via Bilateral Diffusion

Removing Structured Noise with Diffusion Models

Deep Learning Model for Ultrafast Multifrequency Optical Property Extractions for Spatial Frequency Domain Imaging.

DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

Diffusion Posterior Sampling for Nonlinear CT Reconstruction