A Plug-and-Play Method for Guided Multi-contrast MRI Reconstruction based on Content/Style Modeling

Chinmay Rao,Matthias van Osch,Nicola Pezzotti,Jeroen de Bresser,Laurens Beljaards,Jakob Meineke,Elwin de Weerdt,Huangling Lu,Mariya Doneva,Marius Staring

2024-09-20

Abstract:Since multiple MRI contrasts of the same anatomy contain redundant information, one contrast can be used as a prior for guiding the reconstruction of an undersampled subsequent contrast. To this end, several learning-based guided reconstruction methods have been proposed. However, two key challenges remain - (a) the requirement of large paired training datasets and (b) the lack of intuitive understanding of the model's internal representation and utilization of the shared information. We propose a modular two-stage approach for guided reconstruction, addressing these challenges. A content/style model of two-contrast image data is learned in a largely unpaired manner and is subsequently applied as a plug-and-play operator in iterative reconstruction. The disentanglement of content and style allows explicit representation of contrast-independent and contrast-specific factors. Based on this, incorporating prior information into the reconstruction reduces to simply replacing the aliased reconstruction content with clean content derived from the reference scan. We name this novel approach PnP-MUNIT. Various aspects like interpretability and convergence are explored via simulations. Furthermore, its practicality is demonstrated on the NYU fastMRI DICOM dataset and two in-house raw datasets, obtaining up to 32.6% more acceleration over learning-based non-guided reconstruction for a given SSIM. In a radiological task, PnP-MUNIT allowed 33.3% more acceleration over clinical reconstruction at diagnostic quality.

Image and Video Processing,Computer Vision and Pattern Recognition,Medical Physics

What problem does this paper attempt to address?

This paper attempts to solve two key problems in multi - contrast magnetic resonance imaging (MRI) reconstruction:

1. **The need for large - scale paired training data sets**: Most existing learning - based guided reconstruction methods require a large amount of paired training data, which is difficult to obtain in the actual clinical environment. This requirement limits the effective use of these methods in real - world clinical MR data.
2. **Lack of an intuitive understanding of the internal representation of the model and the utilization of shared information**: Existing methods cannot clearly explain how to represent and utilize the shared structural information between images, resulting in insufficient transparency and interpretability.

To solve these problems, the authors propose a modular two - stage method called PnP - MUNIT. This method separates contrast - independent and contrast - specific information through content / style modeling and applies it to iterative reconstruction, thereby achieving more effective guided reconstruction.

### Specific problem description

In multi - contrast MRI reconstruction, since different - contrast images of the same anatomical structure contain redundant information, one contrast image can be used as prior information to guide the reconstruction of another undersampled contrast image. However, existing methods face the following challenges:

- **The need for large - scale paired training data**: A large amount of paired training data is required to train the model, which is difficult to meet in practical applications.
- **Lack of understanding of the internal representation of the model**: It is difficult to intuitively understand how the model represents and utilizes shared information, resulting in poor interpretability of the model.

### Solution

The PnP - MUNIT method proposed by the authors mainly solves the above two problems:

1. **Unpaired training and fine - tuning strategy**:
- Use unpaired data for pre - training to separate contrast - independent (content) and contrast - specific (style) information.
- Use a small amount of paired data for fine - tuning to improve the accuracy of content representation.

2. **Content restoration operator**:
- Define a content restoration operator that can replace the distorted content in the undersampled image with the clean content from the reference image during the reconstruction process.

3. **PnP - MUNIT algorithm**:
- Embed content / style modeling into the iterative reconstruction framework to form a modular algorithm, combining the advantages of the plug - and - play (PnP) method and the semantic interpretability of content / style decomposition.

4. **Experimental verification**:
- Through extensive experiments, the interpretability and convergence of PnP - MUNIT are explored, and its application potential on actual raw data and its clinical practicality in radiology tasks are demonstrated.

### Mathematical formulas

To better understand the working principle of PnP - MUNIT, the following are several key formulas:

- **CS reconstruction optimization problem**:
\[
\min_{x} \|Ax - y\|_2^2+\lambda\|\Psi x\|_1
\]
where \(A\) is the MRI forward operator, \(y\) is the acquired k - space sample, \(\Psi\) is the sparse transform, and \(\lambda\) is the regularization strength.

- **ISTAL algorithm update steps**:
\[
r_k\leftarrow\Psi^H\text{soft}(\Psi x_{k - 1};\lambda)
\]
\[
x_k\leftarrow r_k-\eta A^H(A r_k - y)
\]

- **Content restoration operator**:
\[
r_k^2\leftarrow g_M(x_{k - 1}^2;c)
\]
where \(g_M\) is the content restoration operator based on the content / style model \(M\), and \(c\) is the content extracted from the reference image.

- **Content consistency update**:
\[
c_k\leftarrow c_{k - 1}-\gamma\nabla_c\|A G_2(c_{k - 1},E_s^2(x_k^2)) - y\|_2^2
\]

Through these methods and formulas, PnP - MUNIT effectively solves the key problems in multi - contrast MRI reconstruction and improves the reconstruction quality.