Scalable network reconstruction in subquadratic time

Tiago P. Peixoto

2024-05-08

Abstract:Network reconstruction consists in determining the unobserved pairwise couplings between $N$ nodes given only observational data on the resulting behavior that is conditioned on those couplings -- typically a time-series or independent samples from a graphical model. A major obstacle to the scalability of algorithms proposed for this problem is a seemingly unavoidable quadratic complexity of $\Omega(N^2)$, corresponding to the requirement of each possible pairwise coupling being contemplated at least once, despite the fact that most networks of interest are sparse, with a number of non-zero couplings that is only $O(N)$. Here we present a general algorithm applicable to a broad range of reconstruction problems that significantly outperforms this quadratic baseline. Our algorithm relies on a stochastic second neighbor search (Dong et al., 2011) that produces the best edge candidates with high probability, thus bypassing an exhaustive quadratic search. If we rely on the conjecture that the second-neighbor search finishes in log-linear time (Baron & Darling, 2020; 2022), we demonstrate theoretically that our algorithm finishes in subquadratic time, with a data-dependent complexity loosely upper bounded by $O(N^{3/2}\log N)$, but with a more typical log-linear complexity of $O(N\log^2N)$. In practice, we show that our algorithm achieves a performance that is many orders of magnitude faster than the quadratic baseline -- in a manner consistent with our theoretical analysis -- allows for easy parallelization, and thus enables the reconstruction of networks with hundreds of thousands and even millions of nodes and edges.

Data Structures and Algorithms,Machine Learning,Data Analysis, Statistics and Probability,Computation

What problem does this paper attempt to address?

This paper attempts to solve a major obstacle in network reconstruction, namely the problem of algorithm complexity. Specifically, most existing network reconstruction algorithms face a time complexity of at least \(O(N^2)\), where \(N\) is the number of nodes in the network. This complexity limits the efficiency and practicality of the algorithms when dealing with large - scale networks, especially in practical applications where the amount of data is often very large.

The paper proposes a new general - purpose algorithm that can significantly reduce the time complexity of network reconstruction to the subquadratic level. By using a method based on random second - nearest - neighbor search, the algorithm can find the best edge candidates without performing an exhaustive search, thus avoiding the problem of having to check every possible edge in traditional methods. Theoretical analysis shows that the time complexity of this algorithm can reach \(O(N^{3/2}\log N)\), but in practical applications it is usually closer to \(O(N\log^2 N)\).

In addition, this algorithm also has the property of being easy to parallelize, enabling it to handle large - scale networks containing hundreds of thousands or even millions of nodes and edges. This not only greatly improves the efficiency of network reconstruction, but also provides new tools and methods for large - scale network analysis.