Abstract:Background: High-throughput sequencing has rapidly become an essential part of precision cancer medicine. But validating results obtained from analyzing and interpreting genomic data remains a rate-limiting factor. The gold standard, of course, remains manual validation by expert panels, which is not without its weaknesses, namely high costs in both funding and time as well as the necessarily selective nature of manual validation. But it may be possible to develop more economical, complementary means of validation. In this study we employed four synthetic data sets (variants with known mutations spiked into specific genomic locations) of increasing complexity to assess the sensitivity, specificity, and balanced accuracy of five open-source variant callers: FreeBayes v1.0, VarDict v11.5.1, MuTect v1.1.7, MuTect2, and MuSE v1.0rc. FreeBayes, VarDict, and MuTect were run in bcbio-next gen, and the results were integrated into a single Ensemble call set. The known mutations provided a level of "ground truth" against which we evaluated variant-caller performance. We further facilitated the comparison and evaluation by segmenting the whole genome into 10,000,000 base-pair fragments which yielded 316 segments. Results: Differences among the numbers of true positives were small among the callers, but the numbers of false positives varied much more when the tools were used to analyze sets one through three. Both FreeBayes and VarDict produced strikingly more false positives than did the others, although VarDict, somewhat paradoxically also produced the highest number of true positives. The Ensemble approach yielded results characterized by higher specificity and balanced accuracy and fewer false positives than did any of the five tools used alone. Sensitivity and specificity, however, declined for all five callers as the complexity of the data sets increased, but we did not uncover anything more than limited, weak correlations between caller performance and certain DNA structural features: gene density and guanine-cytosine content. Altogether, MuTect2 performed the best among the callers tested, followed by MuSE and MuTect. Conclusions: Spiking data sets with specific mutations -single-nucleotide variations (SNVs), single-nucleotide polymorphisms (SNPs), or structural variations (SVs) in this study-at known locations in the genome provides an effective and economical way to compare data analyzed by variant callers with ground truth. The method constitutes a viable alternative to the prolonged, expensive, and noncomprehensive assessment by expert panels. It should be further developed and refined, as should other comparatively "lightweight" methods of assessing accuracy. Given that the scientific community has not yet established gold standards for validating NGS-related technologies such as variant callers, developing multiple alternative means for verifying variant-caller accuracy will eventually lead to the establishment of higher-quality standards than could be achieved by prematurely limiting the range of innovative methods explored by members of the community.

A side-by-side comparison of variant function measurements using deep mutational scanning and base editing

Joint genotypic and phenotypic outcome modeling improves base editing variant effect quantification

Towards optimizing diversifying base editors for high-throughput studies of single-nucleotide variants

Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations

Activity-based selection for enhanced base editor mutational scanning

A comprehensive benchmark for multiple highly efficient base editors with broad targeting scope

Annotation and Evaluation of Base Editing Outcomes in Multiple Cell Types Using CRISPRbase.

A benchmarked, high-efficiency prime editing platform for multiplexed dropout screening

EditR: A novel base editing quantification software using Sanger sequencing

Comprehensive evaluation and prediction of editing outcomes for near-PAMless adenine and cytosine base editors

EditR: A Method to Quantify Base Editing from Sanger Sequencing

Expanding the Genome-Targeting Scope and the Site Selectivity of High-Precision Base Editors

OUP accepted manuscript

Improving Editing Efficiency for the Sequences with NGH PAM Using Xcas9-Derived Base Editors.

CHANGE-seq-BE enables simultaneously sensitive and unbiased profiling of base editor genome-wide activity

Scalable assessment of genome editing off-targets associated with genetic variants

Dissecting quantitative trait nucleotides by saturation genome editing

Long-read sequencing for fast and robust identification of correct genome-edited alleles: PCR-based and Cas9 capture methods

Comparing the performance of selected variant callers using synthetic data and genome segmentation

Prime editing in mice reveals the essentiality of a single base in driving tissue-specific gene expression

Deep learning models to predict the editing efficiencies and outcomes of diverse base editors