Abstract:The availability of high-throughput parallel methods for sequencing microbial communities is increasing our knowledge of the microbial world at an unprecedented rate. Though most attention has focused on determining lower-bounds on the alpha-diversity i.e. the total number of different species present in the environment, tight bounds on this quantity may be highly uncertain because a small fraction of the environment could be composed of a vast number of different species. To better assess what remains unknown, we propose instead to predict the fraction of the environment that belongs to unsampled classes. Modeling samples as draws with replacement of colored balls from an urn with an unknown composition, and under the sole assumption that there are still undiscovered species, we show that conditionally unbiased predictors and exact prediction intervals (of constant length in logarithmic scale) are possible for the fraction of the environment that belongs to unsampled classes. Our predictions are based on a Poissonization argument, which we have implemented in what we call the Embedding algorithm. In fixed i.e. non-randomized sample sizes, the algorithm leads to very accurate predictions on a sub-sample of the original sample. We quantify the effect of fixed sample sizes on our prediction intervals and test our methods and others found in the literature against simulated environments, which we devise taking into account datasets from a human-gut and -hand microbiota. Our methodology applies to any dataset that can be conceptualized as a sample with replacement from an urn. In particular, it could be applied, for example, to quantify the proportion of all the unseen solutions to a binding site problem in a random RNA pool, or to reassess the surveillance of a certain terrorist group, predicting the conditional probability that it deploys a new tactic in a next attack.

Optimal Estimation of Bacterial Growth Rates Based on Permuted Monotone Matrix

Multi-sample Estimation of Bacterial Composition Matrix in Metagenomics Data

Matrix Reordering for Noisy Disordered Matrices: Optimality and Computationally Efficient Algorithms

Optimal Structured Matrix Approximation for Robustness to Incomplete Biosequence Data

Estimating maximal microbial growth rates from cultures, metagenomes, and single cells via codon usage patterns

Bayesian biclustering for microbial metagenomic sequencing data via multinomial matrix factorization

A Flexible Zero-Inflated Poisson-Gamma model with application to microbiome read counts

A Zero-Inflated Logistic Normal Multinomial Model for Extracting Microbial Compositions

Robust Covariance Estimation for High-dimensional Compositional Data with Application to Microbial Communities Analysis

Optimal rates for ranking a permuted isotonic matrix in polynomial time

A new robust covariance matrix estimation for high‐dimensional microbiome data

Multi-sample estimation of centered log-ratio matrix in microbiome studies

Estimating Fold Changes from Partially Observed Outcomes with Applications in Microbial Metagenomics

Improved maximum growth rate prediction from microbial genomes by integrating phylogenetic information

Covariance Matrix Estimation for High-Throughput Biomedical Data with Interconnected Communities

Accurate Profiling of Microbial Communities from Massively Parallel Sequencing using Convex Optimization

Prediction analysis for microbiome sequencing data

Extrapolation of Urn Models via Poissonization: Accurate Measurements of the Microbial Unknown

Accurate Genome Relative Abundance Estimation Based on Shotgun Metagenomic Reads

A probabilistic analysis of shotgun sequencing for metagenomics

Power and sample-size estimation for microbiome studies using pairwise distances and PERMANOVA