Abstract:The increasing availability of cloud computing services for science has changed the way scientific code can be developed, deployed, and run. Many modern scientific workflows are capable of running on cloud computing resources. Consequently, there is an increasing interest in the scientific computing community in methods, tools, and implementations that enable moving an application to the cloud and simplifying the process, and decreasing the time to meaningful scientific results. In this paper, we have applied the concepts of containerization for portability and multi-cloud automated deployment with industry-standard tools to three scientific workflows. We show how our implementations provide reduced complexity to portability of both the applications themselves, and their deployment across private and public clouds. Each application has been packaged in a Docker container with its dependencies and necessary environment setup for production runs. Terraform and Ansible have been used to automate the provisioning of compute resources and the deployment of each scientific application in a Multi-VM cluster. Each application has been deployed on the AWS and Aristotle Cloud Federation platforms. Variation in data management constraints, Multi-VM MPI communication, and embarrassingly parallel instance deployments were all explored and reported on. We thus present a sample of scientific workflows that can be simplified using the tools and our proposed implementation to deploy and run in a variety of cloud environments.

Distributed-Something: scripts to leverage AWS storage and computing for distributed workflows at scale

Reproducible and Portable Workflows for Scientific Computing and HPC in the Cloud

Leveraging DevOps for Scientific Computing

A GPU-accelerated Molecular Docking Workflow with Kubernetes and Apache Airflow

BigDataScript: a scripting language for data pipelines

Toward Real-time Analysis of Experimental Science Workloads on Geographically Distributed Supercomputers

AiiDAlab -- an ecosystem for developing, executing, and sharing scientific workflows

An introduction to Docker for reproducible research

Octopus: Experiences with a Hybrid Event-Driven Architecture for Distributed Scientific Computing

Sustainability and Reproducibility via Containerized Computing

Adaptive Container Service: a New Paradigm for Robust and Optimized Bioinformatics Workflow Deployment in the Cloud.

Reproducible Workflow on a Public Cloud for Computational Fluid Dynamics

Evaluating Distributed Execution of Workloads

Collaboration Online: The Example of Distributed Computing

ProvDeploy: Provenance-oriented Containerization of High Performance Computing Scientific Workflows

Bringing numerous methods for expression and promoter analysis to a public cloud computing service

Dflow, a Python framework for constructing cloud-native AI-for-Science workflows

Distributed computing of Seismic Imaging Algorithms

Scientific Workflow Applications on Amazon EC2

A Distributed Parallel Computing Environment for Bioinformatics Problems

Toward Interlanguage Parallel Scripting for Distributed-Memory Scientific Computing