Abstract:Container technologies such as Docker have become a crucial component of many software industry practices especially those pertaining to reproducibility and portability. The containerization philosophy has influenced the scientific computing community, which has begun to adopt - and even develop - container technologies (such as Singularity). Leveraging containers for scientific software often poses challenges distinct from those encountered in industry, and requires different methodologies. This is especially true for HPC. With an increasing number of options for HPC in the cloud (including NSF-funded cloud projects), there is strong motivation to seek solutions that provide flexibility to develop and deploy scientific software on a variety of computational infrastructures in a portable and reproducible way. The flexibility offered by cloud services enables virtual HPC clusters that scale on-demand, and the Cyberinfrastructure Resource Integration team in the XSEDE project has developed a set of tools which provides scalable infrastructure in the cloud. We now present a solution which uses the Nix package manager in an MPI-capable Docker container that is converted to Singularity. It provides consistent installations, dependencies, and environments in each image that are reproducible and portable across scientific computing infrastructures. We demonstrate the utility of these containers with cluster benchmark runs in a self-scaling virtual cluster using the Slurm scheduler deployed in the Jetstream and Aristotle Red Cloud OpenStack clouds. We conclude this technique is useful as a template for scientific software application containers to be used in the XSEDE compute environment, other Singularity HPC environments, and cloud computing environments.

Batch System Prototype Using Containers

Houdini's Escape

Docker-Enabled Scalable Parallel MLFMA System for RCS Evaluation

Sustainability and Reproducibility via Containerized Computing

Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing

Containers for portable, productive and performant scientific computing

JINR Container Distribution Service

Container solutions for HPC Systems: A Case Study of Using Shifter on Blue Waters

Improving the Effective Utilization of Supercomputer Resources by Adding Low-Priority Containerized Jobs

A containerized analytics framework for data and compute-intensive pipeline applications

Survey of adaptive containerization architectures for HPC

Performance evaluation of container-based virtualization for high performance computing environments

Container orchestration on HPC systems through Kubernetes

Carbon Containers: A System-level Facility for Managing Application-level Carbon Emissions

Engineering and Experimentally Benchmarking a Container-based Edge Computing System

Evaluating integration and performance of containerized climate applications on a Hewlett Packard Enterprise Cray system

Batch-oriented software appliances

Kub: Enabling Elastic HPC Workloads on Containerized Environments

Containerisation for High Performance Computing Systems: Survey and Prospects

Reproducible and Portable Workflows for Scientific Computing and HPC in the Cloud

Implementation and evaluation of a container management platform on Docker: Hadoop deployment as an example