Abstract:ABSTRACT Rapid technological advancements in sequencing technologies allow producing cost effective and high volume sequencing data. Processing this data for real-time clinical diagnosis is potentially time-consuming if done on a single computing node. This work presents a complete variant calling workflow, implemented using the Message Passing Interface (MPI) to leverage the benefits of high bandwidth interconnects. This solution (GenMPI) is portable and flexible, meaning it can be deployed to any private or public cluster/cloud infrastructure. Any alignment or variant calling application can be used with minimal adaptation. To achieve high performance, compressed input data can be streamed in parallel to alignment applications while uncompressed data can use internal file seek functionality to eliminate the bottleneck of streaming input data from a single node. Alignment output can be directly stored in multiple chromosome-specific SAM files or a single SAM file. After alignment, a distributed queue using MPI RMA (Remote Memory Access) atomic operations is created for sorting, indexing, marking of duplicates (if necessary) and variant calling applications. We ensure the accuracy of variants as compared to the original single node methods. We also show that for 300x coverage data, alignment scales almost linearly up to 64 nodes (8192 CPU cores). Overall, this work outperforms existing big data based workflows by a factor of two and is almost 20% faster than other MPI-based implementations for alignment without any extra memory overheads. Sorting, indexing, duplicate removal and variant calling is also scalable up to 8 nodes cluster. For pair-end short-reads (Illumina) data, we integrated the BWA-MEM aligner and three variant callers (GATK HaplotypeCaller, DeepVariant and Octopus), while for long-reads data, we integrated the Minimap2 aligner and three different variant callers (DeepVariant, DeepVariant with WhatsHap for phasing (PacBio) and Clair3 (ONT)). All codes and scripts are available at: https://github.com/abs-tudelft/gen-mpi

Joint Haplotype Phasing and Genotype Calling of Multiple Individuals Using Haplotype Informative Reads

Genotype Calling And Haplotype Phasing From Next Generation Sequencing Data

Fast and accurate haplotype inference with hidden markov model

A General Approach for Haplotype Phasing across the Full Spectrum of Relatedness

Improving sequence-based genotype calls with linkage disequilibrium and pedigree information

A Fast and Flexible Statistical Model for Large-Scale Population Genotype Data: Applications to Inferring Missing Genotypes and Haplotypic Phase

Integer programming framework for pangenome-based genome inference

Phasing Nanopore Genome Assembly by Integrating Heterozygous Variations and Hi-C Data

WinHAP: an Efficient Haplotype Phasing Algorithm Based on Scalable Sliding Windows.

MaCH: using sequence and genotype data to estimate haplotypes and unobserved genotypes

A statistical framework for SNP calling, mutation discovery, association mapping and population genetical parameter estimation from sequencing data

A fast data-driven method for genotype imputation, phasing and local ancestry inference: MendelImpute.jl

Evaluation of Consensus Strategies for Haplotype Phasing

GenMPI: Cluster Scalable Variant Calling for Short/Long Reads Sequencing Data

Comparison of haplotype inference methods using genotypic data from unrelated individuals.

Haplotype Inference Using A Bayesian Hidden Markov Model

Genotype Calling from Population-Genomic Sequencing Data

SNP Calling Using Genotype Model Selection on High-Throughput Sequencing Data

Accurate Estimation of Haplotype Frequency from Pooled Sequencing Data and Cost-Effective Identification of Rare Haplotype Carriers by Overlapping Pool Sequencing

HiPhase: jointly phasing small, structural, and tandem repeat variants from HiFi sequencing

A Fast Data-Driven Method for Genotype Imputation, Phasing, and Local Ancestry Inference: MendelImpute.jl