Abstract:We derive new time-space tradeoff lower bounds and algorithms for exactly computing statistics of input data, including frequency moments, element distinctness, and order statistics, that are simple to calculate for sorted data. We develop a randomized algorithm for the element distinctness problem whose time T and space S satisfy T in O (n^{3/2}/S^{1/2}), smaller than previous lower bounds for comparison-based algorithms, showing that element distinctness is strictly easier than sorting for randomized branching programs. This algorithm is based on a new time and space efficient algorithm for finding all collisions of a function f from a finite set to itself that are reachable by iterating f from a given set of starting points. We further show that our element distinctness algorithm can be extended at only a polylogarithmic factor cost to solve the element distinctness problem over sliding windows, where the task is to take an input of length 2n-1 and produce an output for each window of length n, giving n outputs in total. In contrast, we show a time-space tradeoff lower bound of T in Omega(n^2/S) for randomized branching programs to compute the number of distinct elements over sliding windows. The same lower bound holds for computing the low-order bit of F_0 and computing any frequency moment F_k, k neq 1. This shows that those frequency moments and the decision problem F_0 mod 2 are strictly harder than element distinctness. We complement this lower bound with a T in O(n^2/S) comparison-based deterministic RAM algorithm for exactly computing F_k over sliding windows, nearly matching both our lower bound for the sliding-window version and the comparison-based lower bounds for the single-window version. We further exhibit a quantum algorithm for F_0 over sliding windows with T in O(n^{3/2}/S^{1/2}). Finally, we consider the computations of order statistics over sliding windows.

An Evaluation of Streaming Algorithms for Distinct Counting Over a Sliding Window

Counting distinct objects over sliding windows.

An Algorithm for Data Stream Sampling Based on Ring Circular Sliding Window Tightly-Coupled with Buffer

Finding Frequent Closed Itemsets in Sliding Window in Linear Time.

Continuous Monitoring of Distributed Data Streams over a Time-Based Sliding Window

Flattened Exponential Histogram for Sliding Window Queries over Data Streams

Sliding Window-based Approximate Triangle Counting over Streaming Graphs with Duplicate Edges

Pay for a Sliding Bloom Filter and Get Counting, Distinct Elements, and Entropy for Free

Optimal streaming and tracking distinct elements with high probability

Sliding Window Top-K Monitoring over Distributed Data Streams

Maintaining Aggregate Views over Streams

Sliding-Window Probabilistic Threshold Aggregate Queries on Uncertain Data Streams

Novel structures for counting frequent items in time decayed streams

Interval Selection in Sliding Windows

Quality-driven processing of sliding window aggregates over out-of-order data streams

Element Distinctness, Frequency Moments, and Sliding Windows

SHE: A Generic Framework for Data Stream Mining over Sliding Windows.

Sliding-window Top-K Queries on Uncertain Streams

Sliding window-based approximate triangle counting with bounded memory usage

Sliding window top-k dominating query processing over distributed data streams

Fine-grained Probability Counting for Cardinality Estimation of Data Streams.