Abstract:ABSTRACT Rapid technological advancements in sequencing technologies allow producing cost effective and high volume sequencing data. Processing this data for real-time clinical diagnosis is potentially time-consuming if done on a single computing node. This work presents a complete variant calling workflow, implemented using the Message Passing Interface (MPI) to leverage the benefits of high bandwidth interconnects. This solution (GenMPI) is portable and flexible, meaning it can be deployed to any private or public cluster/cloud infrastructure. Any alignment or variant calling application can be used with minimal adaptation. To achieve high performance, compressed input data can be streamed in parallel to alignment applications while uncompressed data can use internal file seek functionality to eliminate the bottleneck of streaming input data from a single node. Alignment output can be directly stored in multiple chromosome-specific SAM files or a single SAM file. After alignment, a distributed queue using MPI RMA (Remote Memory Access) atomic operations is created for sorting, indexing, marking of duplicates (if necessary) and variant calling applications. We ensure the accuracy of variants as compared to the original single node methods. We also show that for 300x coverage data, alignment scales almost linearly up to 64 nodes (8192 CPU cores). Overall, this work outperforms existing big data based workflows by a factor of two and is almost 20% faster than other MPI-based implementations for alignment without any extra memory overheads. Sorting, indexing, duplicate removal and variant calling is also scalable up to 8 nodes cluster. For pair-end short-reads (Illumina) data, we integrated the BWA-MEM aligner and three variant callers (GATK HaplotypeCaller, DeepVariant and Octopus), while for long-reads data, we integrated the Minimap2 aligner and three different variant callers (DeepVariant, DeepVariant with WhatsHap for phasing (PacBio) and Clair3 (ONT)). All codes and scripts are available at: https://github.com/abs-tudelft/gen-mpi

HTSlib: C library for reading/writing high-throughput sequencing data

Twelve years of SAMtools and BCFtools

vcfpp: a C++ API for rapid processing of the Variant Call Format

Vcflib and tools for processing the VCF variant call format

A spectrum of free software tools for processing the VCF variant call format: vcflib, bio-vcf, cyvcf2, hts-nim and slivar

The Sequence Alignment/Map format and SAMtools

Analysing high-throughput sequencing data in Python with HTSeq 2.0

hictk: blazing fast toolkit to work with .hic and .cool files

GenMPI: Cluster Scalable Variant Calling for Short/Long Reads Sequencing Data

The Scalable Variant Call Representation: Enabling Genetic Analysis Beyond One Million Genomes

HTSSIP: An R package for analysis of high throughput sequencing data from nucleic acid stable isotope probing (SIP) experiments

A statistical framework for SNP calling, mutation discovery, association mapping and population genetical parameter estimation from sequencing data

Sam2bam: High-Performance Framework for NGS Data Preprocessing Tools

Fast and accurate mapping of Complete Genomics reads

TBtools: An Integrative Toolkit Developed for Interactive Analyses of Big Biological Data

SALT: a Fast, Memory-Efficient and SNP-aware Short Read Alignment Tool

Compression of Structured High-Throughput Sequencing Data

Fast and accurate short read alignment with hybrid hash-tree data structure

Sambamba: fast processing of NGS alignment formats

Rhat: Fast Alignment of Noisy Long Reads with Regional Hashing.