Abstract:The International Journal of High Performance Computing Applications, Ahead of Print. Supercomputers have been driving innovations for performance and scaling benefiting several scientific applications for the past few decades. Yet their ecosystems remain virtually unchanged when it comes to integrating distributed data-driven workflows, primarily due to rather rigid access methods and restricted configuration management options. X-as-a-Service model of cloud has introduced, among other features, a developer-centric DevOps approach empowering developers of infrastructure, platform to software artefacts, which, unfortunately contemporary supercomputers still lack. We introduce vClusters (versatile software-defined clusters), which is based on Infrastructure-as-code (IaC) technology. vClusters approach is a unique fusion of HPC and cloud technologies resulting in a software-defined, multi-tenant cluster on a supercomputing ecosystem, that, together with software-defined storage, enable DevOps for complex, data-driven workflows like grid middleware, alongside a classic HPC platform. IaC has been a commonplace in cloud computing, however, it lacked adoption within multi-Petascale ecosystems due to concerns related to performance and interoperability with classic HPC data centres' ecosystems. We present an overview of the Swiss National Supercomputing Centre's flagship Alps ecosystem as an implementation target for vClusters for HPC and data-driven workflows. Alps is based on the Cray-HPE Shasta EX supercomputing platform that includes an IaC compliant, microservices architecture (MSA) management system, which we leverage for demonstrating vClusters usage for our diverse operational workflows. We provide implementation details of two operational vClusters platforms: a classic HPC platform that is used predominantly by hundreds of users running thousands of large-scale numerical simulations batch jobs; and a widely used, data-intensive, Grid computing middleware platform used for CERN Worldwide LHC Computing Grid (WLCG) operations. The resulting solution showcases reuse and reduction of common configuration recipes across vCluster implementations, minimising operational change management overheads while introducing flexibility for managing artefacts for DevOps required by diverse workflows.

OpenCluster: A Flexible Distributed Computing Framework for Astronomical Data Processing

A hybrid architecture for astronomical computing

A Communication Efficient and Scalable Distributed Data Mining for the Astronomical Data

A generic framework for the development of geospatial processing pipelines on clusters

CLUSTER-BASED OCEAN REMOTE SENSING IMAGE FUSION PARALLEL COMPUTING STRATEGY

Study of High Performance Processing Platform Architecture in Remote Sensing

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

The Astronomy Commons Platform: A Deployable Cloud-Based Analysis Platform for Astronomy

Data Mining Techniques on Astronomical Spectra Data - I. Clustering Analysis

Xorbits: Automating Operator Tiling for Distributed Data Science

The Open Connectome Project Data Cluster: Scalable Analysis and Vision for High-Throughput Neuroscience

High-Performance Data Storage for Large Solar Telescope Using Distributed File System

AstroCloud: A Distributed Cloud Computing and Application Platform for Astronomy

A distributed data processing scheme based on Hadoop for synchrotron radiation experiments

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

The Research and Implementation of MUSER CLEAN Algorithm Based on OpenCL

Low-cost high performance distributed data storage for multi-channel observations

Jcluster: an Efficient Java Parallel Environment on a Large-Scale Heterogeneous Cluster.

Versatile software-defined HPC and cloud clusters on Alps supercomputer for diverse workflows

NO2: Speeding Up Parallel Processing of Massive Compute-Intensive Tasks

OpenArray V1.0: a Simple Operator Library for the Decoupling of Ocean Modeling and Parallel Computing