Abstract:Abstract Background Accurate variant detection in the coding regions of the human genome is a key requirement for molecular diagnostics of Mendelian disorders. Efficiency of variant discovery from next-generation sequencing (NGS) data depends on multiple factors, including reproducible coverage biases of NGS methods and the performance of read alignment and variant calling software. Although variant caller benchmarks are published constantly, no previous publications have leveraged the full extent of available gold standard whole-genome (WGS) and whole-exome (WES) sequencing datasets. Results In this work, we systematically evaluated the performance of 4 popular short read aligners (Bowtie2, BWA, Isaac, and Novoalign) and 9 novel and well-established variant calling and filtering methods (Clair3, DeepVariant, Octopus, GATK, FreeBayes, and Strelka2) using a set of 14 “gold standard” WES and WGS datasets available from Genome In A Bottle (GIAB) consortium. Additionally, we have indirectly evaluated each pipeline’s performance using a set of 6 non-GIAB samples of African and Russian ethnicity. In our benchmark, Bowtie2 performed significantly worse than other aligners, suggesting it should not be used for medical variant calling. When other aligners were considered, the accuracy of variant discovery mostly depended on the variant caller and not the read aligner. Among the tested variant callers, DeepVariant consistently showed the best performance and the highest robustness. Other actively developed tools, such as Clair3, Octopus, and Strelka2, also performed well, although their efficiency had greater dependence on the quality and type of the input data. We have also compared the consistency of variant calls in GIAB and non-GIAB samples. With few important caveats, best-performing tools have shown little evidence of overfitting. Conclusions The results show surprisingly large differences in the performance of cutting-edge tools even in high confidence regions of the coding genome. This highlights the importance of regular benchmarking of quickly evolving tools and pipelines. We also discuss the need for a more diverse set of gold standard genomes that would include samples of African, Hispanic, or mixed ancestry. Additionally, there is also a need for better variant caller assessment in the repetitive regions of the coding genome.

The Platinum Pedigree: A long-read benchmark for genetic variants

Curated variation benchmarks for challenging medically relevant autosomal genes

Closing the gap: Solving complex medically relevant genes at scale

A High-Quality Human Reference Panel Reveals the Complexity and Distribution of Genomic Structural Variants

Multi-platform Discovery of Haplotype-Resolved Structural Variation in Human Genomes

A familial, telomere-to-telomere reference for human de novo mutation and recombination from a four-generation pedigree

High-coverage nanopore sequencing of samples from the 1000 Genomes Project to build a comprehensive catalog of human genetic variation

Unveiling novel genetic variants in 370 challenging medically relevant genes using the long read sequencing data of 41 samples from 19 global populations

Enhancing SNV identification in whole-genome sequencing data through the incorporation of known genetic variants into the minimap2 index

Nanopore sequencing of 1000 Genomes Project samples to build a comprehensive catalog of human genetic variation

Systematic benchmark of state-of-the-art variant calling pipelines identifies major factors affecting accuracy of coding sequence variant discovery

The Great Genotyper: A Graph-Based Method for Population Genotyping of Small and Structural Variants

Assessing structural variation in a personal genome—towards a human reference diploid genome

Analysis and benchmarking of small and large genomic variants across tandem repeats

Benchmarking variant callers in next-generation and third-generation sequencing analysis

Generative Haplotype Prediction Outperforms Statistical Methods for Small Variant Detection in NGS Data

Rare coding variant analysis for human diseases across biobanks and ancestries

Characterizing the Genetic Polymorphisms in 370 Challenging Medically Relevant Genes Using Long-Read Sequencing Data from 41 Human Individuals among 19 Global Populations

A deep catalogue of protein-coding variation in 983,578 individuals

Benchmarking reveals superiority of deep learning variant callers on bacterial nanopore sequence data