Abstract:Estimating the number of distinct values (NDV) in a column is useful for many tasks in database systems, such as columnstore compression and data profiling. In this work, we focus on how to derive accurate NDV estimations from random (online/offline) samples. Such efficient estimation is critical for tasks where it is prohibitive to scan the data even once. Existing sample-based estimators typically rely on heuristics or assumptions and do not have robust performance across different datasets as the assumptions on data can easily break. On the other hand, deriving an estimator from a principled formulation such as maximum likelihood estimation is very challenging due to the complex structure of the formulation. We propose to formulate the NDV estimation task in a supervised learning framework, and aim to learn a model as the estimator. To this end, we need to answer several questions: i) how to make the learned model workload agnostic; ii) how to obtain training data; iii) how to perform model training. We derive conditions of the learning framework under which the learned model is workload agnostic, in the sense that the model/estimator can be trained with synthetically generated training data, and then deployed into any data warehouse simply as, e.g., user-defined functions (UDFs), to offer efficient (within microseconds on CPU) and accurate NDV estimations for unseen tables and workloads. We compare the learned estimator with the state-of-the-art sample-based estimators on nine real-world datasets to demonstrate its superior estimation accuracy. We publish our code for training data generation, model training, and the learned estimator online for reproducibility.

A Learning-Based Approach to Estimate Statistics of Operators in Continuous Queries: a Case Study.

Self-Learning Histograms for Changing Workloads.

Automatic Statistics Collection In Oscar Dbms

Learning to be a Statistician: Learned Estimator for Number of Distinct Values

Expressing And Optimizing Similarity-Based Queries In Sql

Database Native Approximate Query Processing Based on Machine-Learning

Operator Learning: Algorithms and Analysis

Learned Cardinality Estimation: A Design Space Exploration and A Comparative Evaluation

Approximate continuous querying over distributed streams

Contextual Ranking of Database Querying Results: A Statistical Approach

An approximate operator-based learning method for the numerical solutions of stochastic differential equations

Mining and using coverage and overlap statistics for data integration

Scalable Statistical Inference in Non-parametric Least Squares

Learned Optimizer for Online Approximate Query Processing in Data Exploration

COOOL: A Learning-To-Rank Approach for SQL Hint Recommendations

LAQP: Learning-based Approximate Query Processing

Mining Source Coverage Statistics for Data Integration

Aggregate Estimations over Location Based Services

A Note On Operator-Level Query Execution Cost Modeling

"What makes my queries slow?": Subgroup Discovery for SQL Workload Analysis

Continually Evaluating Similarity-Based Pattern Queries on a Streaming Time Series