Towards a Dynamic Composability Approach for using Heterogeneous Systems in Remote Sensing
Ilkay Altintas,Ismael Perez,Dmitry Mishin,Adrien Trouillaud,Christopher Irving,John Graham,Mahidhar Tatineni,Thomas DeFanti,Shawn Strande,Larry Smarr,Michael L. Norman
DOI: https://doi.org/10.48550/arXiv.2211.06918
2022-11-13
Abstract:Influenced by the advances in data and computing, the scientific practice increasingly involves machine learning and artificial intelligence driven methods which requires specialized capabilities at the system-, science- and service-level in addition to the conventional large-capacity supercomputing approaches. The latest distributed architectures built around the composability of data-centric applications led to the emergence of a new ecosystem for container coordination and integration. However, there is still a divide between the application development pipelines of existing supercomputing environments, and these new dynamic environments that disaggregate fluid resource pools through accessible, portable and re-programmable interfaces. New approaches for dynamic composability of heterogeneous systems are needed to further advance the data-driven scientific practice for the purpose of more efficient computing and usable tools for specific scientific domains. In this paper, we present a novel approach for using composable systems in the intersection between scientific computing, artificial intelligence (AI), and remote sensing domain. We describe the architecture of a first working example of a composable infrastructure that federates Expanse, an NSF-funded supercomputer, with Nautilus, a Kubernetes-based GPU geo-distributed cluster. We also summarize a case study in wildfire modeling, that demonstrates the application of this new infrastructure in scientific workflows: a composed system that bridges the insights from edge sensing, AI and computing capabilities with a physics-driven simulation.
Distributed, Parallel, and Cluster Computing,Artificial Intelligence