Abstract:The site-to-site variability in species composition, known as β-diversity, is crucial to understanding spatiotemporal patterns of species diversity and the mechanisms controlling community composition and structure. However, quantifying β-diversity in microbial ecology using sequencing-based technologies is a great challenge because of a high number of sequencing errors, bias, and poor reproducibility and quantification. Herein, based on general sampling theory, a mathematical framework is first developed for simulating the effects of random sampling processes on quantifying β-diversity when the community size is known or unknown. Also, using an analogous ball example under Poisson sampling with limited sampling efforts, the developed mathematical framework can exactly predict the low reproducibility among technically replicate samples from the same community of a certain species abundance distribution, which provides explicit evidences of random sampling processes as the main factor causing high percentages of technical variations. In addition, the predicted values under Poisson random sampling were highly consistent with the observed low percentages of operational taxonomic unit (OTU) overlap (<30% and <20% for two and three tags, respectively, based on both Jaccard and Bray-Curtis dissimilarity indexes), further supporting the hypothesis that the poor reproducibility among technical replicates is due to the artifacts associated with random sampling processes. Finally, a mathematical framework was developed for predicting sampling efforts to achieve a desired overlap among replicate samples. Our modeling simulations predict that several orders of magnitude more sequencing efforts are needed to achieve desired high technical reproducibility. These results suggest that great caution needs to be taken in quantifying and interpreting β-diversity for microbial community analysis using next-generation sequencing technologies. IMPORTANCE Due to the vast diversity and uncultivated status of the majority of microorganisms, microbial detection, characterization, and quantitation are of great challenge. Although large-scale metagenome sequencing technology such as PCR-based amplicon sequencing has revolutionized the studies of microbial communities, it suffers from several inherent drawbacks, such as a high number of sequencing errors, biases, poor quantitation, and very high percentages of technical variations, which could greatly overestimate microbial biodiversity. Based on general sampling theory, this study provided the first explicit evidence to demonstrate the importance of random sampling processes in estimating microbial β-diversity, which has not been adequately recognized and addressed in microbial ecology. Since most ecological studies are involved in random sampling, the conclusions learned from this study should also be applicable to other ecological studies in general. In summary, the results presented in this study should have important implications for examining microbial biodiversity to address both basic theoretical and applied management questions.

Sample pooling inflates error rates in between-sample comparisons: an empirical investigation of the statistical properties of count-based data

Effect of Pooling Samples on the Efficiency of Comparative Studies Using Microarrays.

Improving population-specific allele frequency estimates by adapting supplemental data: an empirical Bayes approach

Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation

Adjusting Batch Effects in Microarray Expression Data Using Empirical Bayes Methods.

Biases and errors on allele frequency estimation and disease association tests of next-generation sequencing of pooled samples.

Bagged Empirical Null p-values: A Method to Account for Model Uncertainty in Large Scale Inference

A test metric for assessing single-cell RNA-seq batch correction

Minor Issues Escalated to Critical Levels in Large Samples: A Permutation-Based Fix

How sample heterogeneity can obscure the signal of microbial interactions

Informed Pooled Testing with Quantitative Assays

Thinking points for effective batch correction on biomedical data

Waste Not, Want Not: Why Rarefying Microbiome Data is Inadmissible

Pooling multiple imputations when the sample happens to be the population

Random sampling process leads to overestimation of β-diversity of microbial communities

The Confounding Effects of Population Structure, Genetic Diversity and the Sampling Scheme on the Detection and Quantification of Population Size Changes

Pool samples to efficiently estimate pathogen prevalence dynamics

Using Inverse Probability Bootstrap Sampling to Eliminate Sample Induced Bias in Model Based Analysis of Unequal Probability Samples

A combined efficient design for biomarker data subject to a limit of detection due to measuring instrument sensitivity

Statistical Modeling for Practical Pooled Testing During the COVID-19 Pandemic

Assessing and mitigating batch effects in large-scale omics studies