Abstract:ABSTRACT Rapid technological advancements in sequencing technologies allow producing cost effective and high volume sequencing data. Processing this data for real-time clinical diagnosis is potentially time-consuming if done on a single computing node. This work presents a complete variant calling workflow, implemented using the Message Passing Interface (MPI) to leverage the benefits of high bandwidth interconnects. This solution (GenMPI) is portable and flexible, meaning it can be deployed to any private or public cluster/cloud infrastructure. Any alignment or variant calling application can be used with minimal adaptation. To achieve high performance, compressed input data can be streamed in parallel to alignment applications while uncompressed data can use internal file seek functionality to eliminate the bottleneck of streaming input data from a single node. Alignment output can be directly stored in multiple chromosome-specific SAM files or a single SAM file. After alignment, a distributed queue using MPI RMA (Remote Memory Access) atomic operations is created for sorting, indexing, marking of duplicates (if necessary) and variant calling applications. We ensure the accuracy of variants as compared to the original single node methods. We also show that for 300x coverage data, alignment scales almost linearly up to 64 nodes (8192 CPU cores). Overall, this work outperforms existing big data based workflows by a factor of two and is almost 20% faster than other MPI-based implementations for alignment without any extra memory overheads. Sorting, indexing, duplicate removal and variant calling is also scalable up to 8 nodes cluster. For pair-end short-reads (Illumina) data, we integrated the BWA-MEM aligner and three variant callers (GATK HaplotypeCaller, DeepVariant and Octopus), while for long-reads data, we integrated the Minimap2 aligner and three different variant callers (DeepVariant, DeepVariant with WhatsHap for phasing (PacBio) and Clair3 (ONT)). All codes and scripts are available at: https://github.com/abs-tudelft/gen-mpi

Accelerating Minimap2 for Accurate Long Read Alignment on GPUs

mm2-gb: GPU Accelerated Minimap2 for Long Read DNA Mapping

Efficient end-to-end long-read sequence mapping using minimap2-fpga integrated with hardware accelerated chaining

MinimapR: A Parallel Alignment Tool for the Analysis of Large-Scale Third-Generation Sequencing Data

Accelerating BWA-MEM Read Mapping on GPUs

High-Performance Genomic Analysis Heterogeneous System Using OpenCL

BitMapper2: A GPU-Accelerated All-Mapper Based on the Sparse Q-Gram Index

MaxSSmap: A GPU program for mapping divergent short reads to genomes with the maximum scoring subsequence

AGAThA: Fast and Efficient GPU Acceleration of Guided Sequence Alignment for Long Read Mapping

CUDAMPF++: A Proactive Resource Exhaustion Scheme for Accelerating Homologous Sequence Search on CUDA-enabled GPU

Accelerating whole-genome alignment in the age of complete genome assemblies

gpuPairHMM: High-speed Pair-HMM Forward Algorithm for DNA Variant Calling on GPUs

Accelerating K-mer Frequency Counting with GPU and Non-Volatile Memory

GenMPI: Cluster Scalable Variant Calling for Short/Long Reads Sequencing Data

Parallel Implementation of Mafft on Cuda-Enabled Graphics Hardware

GateSeeder: Near-memory CPU-FPGA Acceleration of Short and Long Read Mapping

Minimap2: pairwise alignment for nucleotide sequences

GPU-accelerated homology search with MMseqs2

GPU Accelerated Biological Sequence Alignment

Accelerating genomic workflows using NVIDIA Parabricks