Abstract:Consider the following gap cycle counting problem in the streaming model: The edges of a 2-regular $n$-vertex graph $G$ are arriving one-by-one in a stream and we are promised that $G$ is a disjoint union of either $k$-cycles or $2k$-cycles for some small $k$; the goal is to distinguish between these two cases using a limited memory. Verbin and Yu [SODA 2011] introduced this problem and showed that any single-pass streaming algorithm solving it requires $n^{1-\Omega(1/k)}$ space. This result and the proof technique behind it—the Boolean Hidden Hypermatching communication problem—has since been used extensively for proving streaming lower bounds for various problems, including approximating MAX-CUT, matching size, property testing, matrix rank and Schatten norms, streaming unique games and CSPs, and many others. Despite its significance and broad range of applications, the lower bound technique of Verbin and Yu comes with a key weakness that is also inherited by all subsequent results: the Boolean Hidden Hypermatching problem is hard only if there is exactly one round of communication and, in fact, can be solved with logarithmic communication in two rounds. Therefore, all streaming lower bounds derived from this problem only hold for single-pass algorithms. Our goal in this paper is to remedy this state-of-affairs. We prove the first multi-pass lower bound for the gap cycle counting problem: Any $p$-pass streaming algorithm that can distinguish between disjoint union of $k$-cycles vs $2k$-cycles—or even $k$-cycles vs one Hamiltonian cycle-requires $n^{1-1/k^{\Omega(1/p)}}$ space. This makes progress on multiple open questions in this line of research dating back to the work of Verbin and Yu. As a corollary of this result and by simple (or even no) modification of prior reductions, we can extend many of previous lower bounds to multi-pass algorithms. For instance, we can now prove that any streaming algorithm that ($1+\varepsilon$) -approximates the value of MAX-CUT, maximum matching size, or rank of an $n$-by-$n$ matrix, requires either $n^{\Omega(1)}$ space or $\Omega(\log({}^{1}\!/\!_{\varepsilon}))$ passes. For all these problems, prior work left open the possibility of even an $O(\log n)$ space algorithm in only two passes.

Lower Bounds for Multi-Pass Processing of Multiple Data Streams

Multi-Pass Graph Streaming Lower Bounds for Cycle Counting, MAX-CUT, Matching Size, and Other Problems

A New Information Complexity Measure for Multi-pass Streaming with Applications

On the Complexity of Processing Massive, Unordered, Distributed Data

An optimization algorithm for data stream joins on multiprocessor

Improved and Partially-Tight Lower Bounds for Message-Passing Implementations of Multiplicity Queues

Parallelizing stateful operators in a distributed stream processing system: how, should you and how much?

Learning state machines via efficient hashing of future traces

Algorithmic Verification of Single-Pass List Processing Programs

The Case of Using Multiple Streams in Streaming

Quasar 3C298: a test-case for meteoritic nanodiamond 3.5 microns emission

Stream Packing for Asynchronous Multi-Context Systems using ASP

Modeling and Analysis of Data Flow Generated by Multi-Sources

Atomic Streaming: A Framework of On-Chip Data Supply System for Task-Parallel Mpsocs

Erlang Model for Multiple Data Streams (Full Version)

Joint Sequential Detection and Isolation for Dependent Data Streams

Reduce to the Max: A Simple Approach for Massive-Scale Privacy-Preserving Collaborative Network Measurements (Extended Version)

Improved Deterministic Streaming Algorithms for Non-monotone Submodular Maximization

The SpaceSaving$\pm$ Family of Algorithms for Data Streams with Bounded Deletions

Duplicate-Insensitive Order Statistics Computation over Data Streams.

Streaming SQL Multi-Way Join Method for Long State Streams