Abstract:Cardinality estimation is one of the most important problems in query optimization. Recently, machine learning-based techniques have been proposed to effectively estimate cardinality, which can be broadly classified into query-driven and data-driven approaches. Query-driven approaches learn a regression model from a query to its cardinality, while data-driven approaches learn a distribution of tuples, select some samples that satisfy a SQL query, and use the data distributions of these selected tuples to estimate the cardinality of the SQL query. As query-driven methods rely on training queries, the estimation quality is not reliable when there are no high-quality training queries, while data-driven methods have no such limitation and have high adaptivity. In this work, we focus on data-driven methods. A good data-driven model should achieve three optimization goals. First, the model needs to capture data dependencies between columns and support large domain sizes (achieving high accuracy). Second, the model should achieve high inference efficiency, because many data samples are needed to estimate the cardinality (achieving low inference latency). Third, the model should not be too large (achieving a small model size). However, existing data-driven methods cannot simultaneously optimize the three goals. To address the limitations, we propose a novel cardinality estimator , which leverages the normalizing flow-based model to learn a continuous joint distribution for relational data. can transform a complex distribution over continuous random variables into a simple distribution (e.g., multivariate normal distribution) and use the probability density to estimate the cardinality for both sequential queries and parallel queries. First, we design a dequantization method to make data more “continuous.” Second, we propose encoding and indexing techniques to handle Like predicates for string data. Third, we propose a Monte Carlo method to estimate the cardinality based on the model. Fourth, we propose a grouping technique to process parallel queries. Fifth, we discuss how to support join queries. Experimental results show that our method significantly outperforms existing approaches in terms of estimation accuracy while keeping similar latency and model size.

Weighted Distinct Sampling: Cardinality Estimation for SPJ Queries

QSketch: An Efficient Sketch for Weighted Cardinality Estimation in Streams

CardSketch: Shift Attention for Network-wide Cardinality Telemetry

NN-based Transformation of Any SQL Cardinality Estimator for Handling DISTINCT, AND, OR and NOT

Learned Cardinality Estimation: A Design Space Exploration and A Comparative Evaluation

Convolution and Cross-Correlation of Count Sketches Enables Fast Cardinality Estimation of Multi-Join Queries

Cardinality Estimation Meets Good-Turing

SAQP++: Bridging the Gap Between Sampling-Based Approximate Query Processing and Aggregate Precomputation.

Adaptive Sampling For Selectivity Estimation In Spatial Database

Cardinality Estimation in DBMS

Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation.

Combining Aggregation and Sampling (Nearly) Optimally for Approximate Query Processing

Cardinality Estimation Using Normalizing Flow

Improving Distributed Similarity Join in Metric Space with Error-bounded Sampling

Sampling-based Estimation of the Number of Distinct Values in Distributed Environment

Leveraging Discarded Samples for Tighter Estimation of Multiple-Set Aggregates

Effective Skyline Cardinality Estimation on Data Streams

Adaptive Cardinality Estimation

Sampling over Union of Joins

Nearly optimal capture-recapture sampling and empirical likelihood weighting estimation for M-estimation with big data

MTS Sketch for Accurate Estimation of Set-Expression Cardinalities from Small Samples