Scalable, reproducible, and cost-effective processing of large-scale medical imaging datasets

Michael E. Kim,Karthik Ramadass,Chenyu Gao,Praitayini Kanakaraj,Nancy R. Newlin,Gaurav Rudravaram,Kurt G. Schilling,Blake E. Dewey,Derek Archer,Timothy J. Hohman,Zhiyuan Li,Shunxing Bao,Bennett A. Landman,Nazirah Mohd Khairi
2024-08-27
Abstract:Curating, processing, and combining large-scale medical imaging datasets from national studies is a non-trivial task due to the intense computation and data throughput required, variability of acquired data, and associated financial overhead. Existing platforms or tools for large-scale data curation, processing, and storage have difficulty achieving a viable cost-to-scale ratio of computation speed for research purposes, either being too slow or too expensive. Additionally, management and consistency of processing large data in a team-driven manner is a non-trivial task. We design a BIDS-compliant method for an efficient and robust data processing pipeline of large-scale diffusion-weighted and T1-weighted MRI data compatible with low-cost, high-efficiency computing systems. Our method accomplishes automated querying of data available for processing and process running in a consistent and reproducible manner that has long-term stability, while using heterogenous low-cost computational resources and storage systems for efficient processing and data transfer. We demonstrate how our organizational structure permits efficiency in a semi-automated data processing pipeline and show how our method is comparable in processing time to cloud-based computation while being almost 20 times more cost-effective. Our design allows for fast data throughput speeds and low latency to reduce the time for data transfer between storage servers and computation servers, achieving an average of 0.60 Gb/s compared to 0.33 Gb/s for using cloud-based processing methods. The design of our workflow engine permits quick process running while maintaining flexibility to adapt to newly acquired data.
Distributed, Parallel, and Cluster Computing,Databases
What problem does this paper attempt to address?