Abstract:The International Journal of High Performance Computing Applications, Ahead of Print. Supercomputers have been driving innovations for performance and scaling benefiting several scientific applications for the past few decades. Yet their ecosystems remain virtually unchanged when it comes to integrating distributed data-driven workflows, primarily due to rather rigid access methods and restricted configuration management options. X-as-a-Service model of cloud has introduced, among other features, a developer-centric DevOps approach empowering developers of infrastructure, platform to software artefacts, which, unfortunately contemporary supercomputers still lack. We introduce vClusters (versatile software-defined clusters), which is based on Infrastructure-as-code (IaC) technology. vClusters approach is a unique fusion of HPC and cloud technologies resulting in a software-defined, multi-tenant cluster on a supercomputing ecosystem, that, together with software-defined storage, enable DevOps for complex, data-driven workflows like grid middleware, alongside a classic HPC platform. IaC has been a commonplace in cloud computing, however, it lacked adoption within multi-Petascale ecosystems due to concerns related to performance and interoperability with classic HPC data centres' ecosystems. We present an overview of the Swiss National Supercomputing Centre's flagship Alps ecosystem as an implementation target for vClusters for HPC and data-driven workflows. Alps is based on the Cray-HPE Shasta EX supercomputing platform that includes an IaC compliant, microservices architecture (MSA) management system, which we leverage for demonstrating vClusters usage for our diverse operational workflows. We provide implementation details of two operational vClusters platforms: a classic HPC platform that is used predominantly by hundreds of users running thousands of large-scale numerical simulations batch jobs; and a widely used, data-intensive, Grid computing middleware platform used for CERN Worldwide LHC Computing Grid (WLCG) operations. The resulting solution showcases reuse and reduction of common configuration recipes across vCluster implementations, minimising operational change management overheads while introducing flexibility for managing artefacts for DevOps required by diverse workflows.

XaaS: Acceleration as a Service to Enable Productive High-Performance Cloud Computing

Elastic Application Container: A Lightweight Approach for Cloud Resource Provisioning

Understanding the Performance and Potential of Cloud Computing for Scientific Applications

Kernel-as-a-Service: A Serverless Programming Model for Heterogeneous Hardware Accelerators

Making Availability as a Service in the Clouds

Cloud Services Enable Efficient AI-Guided Simulation Workflows across Heterogeneous Resources

HPC Cloud for Scientific and Business Applications: Taxonomy, Vision, and Research Challenges

HPC Alongside User-space Kubernetes

Accelerator-as-a-Service in Public Clouds: An Intra-Host Traffic Management View for Performance Isolation in the Wild

VM-MAD: a cloud/cluster software for service-oriented academic environments

HPX with Spack and Singularity Containers: Evaluating Overheads for HPX/Kokkos using an astrophysics application

Versatile software-defined HPC and cloud clusters on Alps supercomputer for diverse workflows

Cloud Scalability Measurement And Testing

A cloud-based TaaS infrastructure with tools for SaaS validation, performance and scalability evaluation

ACaaS: Access Control as a Service for IaaS Cloud

Enabling FPGAs in the cloud.

Software Resource Disaggregation for HPC with Serverless Computing

Arcus: SLO Management for Accelerators in the Cloud with Traffic Shaping

Cloud Computing -- Everything As A Service

Cloud Instance Management and Resource Prediction For Computation-as-a-Service Platforms

Acceleration-as-a-μService: A Cloud-native Monte-Carlo Option Pricing Engine on CPUs, GPUs and Disaggregated FPGAs