Abstract:Deep learning-based Magnetic Resonance (MR) reconstruction methods have focused on generating high-quality images but they often overlook the impact on downstream tasks (e.g., segmentation) that utilize the reconstructed images. Cascading separately trained reconstruction network and downstream task network has been shown to introduce performance degradation due to error propagation and domain gaps between training datasets. To mitigate this issue, downstream task-oriented reconstruction optimization has been proposed for a single downstream task. Expanding this optimization to multi-task scenarios is not straightforward. In this work, we extended this optimization to sequentially introduced multiple downstream tasks and demonstrated that a single MR reconstruction network can be optimized for multiple downstream tasks by deploying continual learning (MOST). MOST integrated techniques from replay-based continual learning and image-guided loss to overcome catastrophic forgetting. Comparative experiments demonstrated that MOST outperformed a reconstruction network without finetuning, a reconstruction network with naïve finetuning, and conventional continual learning methods. This advancement empowers the application of a single MR reconstruction network for multiple downstream tasks. The source code is available at: <a class="link-external link-https" href="https://github.com/SNU-LIST/MOST" rel="external noopener nofollow">this https URL</a>
What problem does this paper attempt to address?
The paper attempts to address the problem of how to optimize a single reconstruction network in magnetic resonance imaging (MRI) reconstruction to adapt to multiple downstream tasks (such as segmentation, classification, etc.) without causing catastrophic forgetting. Specifically, existing methods usually optimize for a single downstream task, requiring multiple reconstruction networks when dealing with multiple downstream tasks, which is inefficient in practical applications. Therefore, this paper proposes a continual learning-based method—MOST (MR reconstruction Optimization for multiple down Stream Tasks via continual learning), which enables a single reconstruction network to optimize multiple downstream tasks through sequential finetuning while preventing significant performance degradation on previous tasks.
### Main Issues
1. **Challenges of Multi-task Optimization**: Traditional single-task optimization methods cannot be directly extended to multi-task scenarios because the data structures and label types of different tasks may vary, making it difficult to form a unified multi-task optimization strategy.
2. **Catastrophic Forgetting**: During sequential finetuning, the network may forget previously learned tasks, leading to performance degradation.
3. **Limitations in Practical Applications**: In practical applications, due to privacy and computational cost constraints, it is impossible to access the complete datasets of all previous tasks.
### Solutions
1. **Continual Learning**: Adopting a replay-based continual learning method, maintaining a fixed-size buffer to store data pairs from previous tasks, thus utilizing these data during the finetuning of subsequent tasks to prevent catastrophic forgetting.
2. **Image-guided Loss**: Introducing image-guided loss when handling downstream tasks that generate images (such as segmentation tasks), enhancing the performance of the reconstruction network by calculating the difference between intermediate reconstructed images and non-aliased images.
3. **Sequential Finetuning**: During the finetuning of each new task, only the reconstruction network is finetuned while the downstream task network remains fixed. After each finetuning, some input-output pairs are stored in the buffer for use in subsequent tasks.
### Experimental Results
Experimental results show that the MOST method performs excellently across multiple downstream tasks, not only improving the performance of downstream tasks but also effectively preventing catastrophic forgetting. Compared to no finetuning, simple finetuning, and traditional continual learning methods, MOST achieves the best performance metrics in all tasks, including Structural Similarity Index (SSIM), Dice Similarity Coefficient (DICE), and Area Under the Curve (AUC).
### Summary
By proposing the MOST method, this paper successfully addresses the problem of optimizing a single network to adapt to multiple downstream tasks in MRI reconstruction, providing an efficient and reliable solution for practical applications.