Distribution of the threshold for the symmetric perceptron

Ashwin Sah,Mehtaab Sawhney

DOI: https://doi.org/10.48550/arXiv.2301.10701

2023-01-25

Probability

Abstract:We derive an explicit distribution for the threshold sequence of the symmetric binary perceptron with Gaussian disorder, proving that the critical window is of constant width.

What problem does this paper attempt to address?

Critical Window of The Symmetric Perceptron

Dylan J. Altschuler

2023-08-09

Abstract:We study the critical window of the symmetric binary perceptron, or equivalently, combinatorial discrepancy. Consider the problem of finding a binary vector $\sigma$ satisfying $\|A\sigma\|_\infty \le K$, where $A$ is an $\alpha n \times n$ matrix with iid Gaussian entries. For fixed $K$, at which densities $\alpha$ is this constraint satisfaction problem (CSP) satisfiable? A sharp threshold was recently established by Perkins and Xu, and Abbe, Li, and Sly , answering this to first order. Namely, for each $K$ there exists an explicit critical density $\alpha_c$ so that for any fixed $\epsilon > 0$, with high probability the CSP is satisfiable for $\alpha n < (\alpha_c - \epsilon ) n$ and unsatisfiable for $\alpha n > (\alpha_c + \epsilon) n$. This corresponds to a bound of $o(n)$ on the size of the critical window. We sharpen these results significantly, as well as provide exponential tail bounds. Our main result is that, perhaps surprisingly, the critical window is actually at most $O(\log n)$. More precisely, with high probability the CSP is satisfiable for $\alpha n < \alpha_c n -O(\log n)$ and unsatisfiable for any $\alpha n > \alpha_c n + \omega(1)$. This implies the symmetric perceptron has nearly the "sharpest possible transition," adding it to a short list of CSP for which the critical window is rigorously known to be of near-constant width.

Probability,Discrete Mathematics,Mathematical Physics,Combinatorics
On the atypical solutions of the symmetric binary perceptron

Damien Barbier,Ahmed El Alaoui,Florent Krzakala,Lenka Zdeborova

DOI: https://doi.org/10.1088/1751-8121/ad3a4a

2024-04-04

Journal of Physics A Mathematical and Theoretical

Abstract:We study the random binary symmetric perceptron problem, focusing on the behavior of rare high-margin solutions. While most solutions are isolated, we demonstrate that these rare solutions are part of clusters of extensive entropy, heuristically corresponding to non-trivial fixed points of an approximate message-passing algorithm. We enumerate these clusters via a local entropy, defined as a Franz-Parisi potential, which we rigorously evaluate using the first and second moment methods in the limit of a small constraint density (corresponding to vanishing margin ) under a certain assumption on the concentration of the entropy. This examination unveils several intriguing phenomena: i) We demonstrate that these clusters have an entropic barrier in the sense that the entropy as a function of the distance from the reference high-margin solution is non-monotone when , while it is monotone otherwise, and that they have an energetic barrier in the sense that there are no solutions at an intermediate distance from the reference solution when . The critical scaling of the margin in corresponds to the one obtained from the earlier work of Gamarnik et al. for the overlap-gap property, a phenomenon known to present a barrier to certain efficient algorithms. ii) We establish using the replica method that the complexity (the logarithm of the number of clusters of such solutions) versus entropy (the logarithm of the number of solutions in the clusters) curves are partly non-concave and correspond to very large values of the Parisi parameter, with the equilibrium being reached when the Parisi parameter diverges.

physics, multidisciplinary, mathematical
Capacity Lower Bound for the Ising Perceptron.

Jian Ding,Nike Sun

DOI: https://doi.org/10.1145/3313276.3316383

2019-01-01

Abstract:We consider the Ising perceptron with gaussian disorder, which is equivalent to the discrete cube {−1,+1}N intersected by M random half-spaces. The perceptron’s capacity is the largest integer MN for which the intersection is nonempty. It is conjectured by Krauth and Mézard (1989) that the (random) ratio MN/N converges in probability to an explicit constant α⋆≐ 0.83. Kim and Roche (1998) proved the existence of a positive constant γ such that γ ≤ MN/N ≤ 1−γ with high probability; see also Talagrand (1999). In this paper we show that the Krauth–Mézard conjecture α⋆ is a lower bound with positive probability, under the condition that an explicit univariate function S(λ) is maximized at λ=0. Our proof is an application of the second moment method to a certain slice of perceptron configurations, as selected by the so-called TAP (Thouless, Anderson, and Palmer, 1977) or AMP (approximate message passing) iteration, whose scaling limit has been characterized by Bayati and Montanari (2011) and Bolthausen (2012). For verifying the condition on S(λ) we outline one approach, which is implemented in the current version using (nonrigorous) numerical integration packages. In a future version of this paper we intend to complete the verification by implementing a rigorous numerical method.
How to escape atypical regions in the symmetric binary perceptron: a journey through connected-solutions states

Damien Barbier

2024-08-12

Abstract:We study the binary symmetric perceptron model, and in particular its atypical solutions. While the solution-space of this problem is dominated by isolated configurations, it is also solvable for a certain range of constraint density $\alpha$ and threshold $\kappa$. We provide in this paper a statistical measure probing sequences of solutions, where two consecutive elements shares a strong overlap. After simplifications, we test its predictions by comparing it to Monte-Carlo simulations. We obtain good agreement and show that connected states with a Markovian correlation profile can fully decorrelate from their initialization only for $\kappa>\kappa_{\rm no-mem.\, state}$ ($\kappa_{\rm no-mem.\, state}\sim \sqrt{0.91\log(N)}$ for $\alpha=0.5$ and $N$ being the dimension of the problem). For $\kappa<\kappa_{\rm no-mem.\, state}$, we show that decorrelated sequences still exist but have a non-trivial correlations profile. To study this regime we introduce an $Ansatz$ for the correlations that we label as the nested Markov chain.

Disordered Systems and Neural Networks,Statistical Mechanics,Mathematical Physics
Scaling Limits for One-Dimensional Long-Range Percolation: Using the Corrector Method

Zhongyang Zhang,Lixin Zhang

DOI: https://doi.org/10.1016/j.spl.2013.06.036

IF: 0.718

2013-01-01

Statistics & Probability Letters

Abstract:In this paper, by using the corrector method we give another proof of the quenched invariance principle for the random walk on the infinite random graph generated by a one-dimensional long-range percolation under the conditions that the connection probability p(1)=1 and the percolation exponent s>2. The key step of the proof is the construction of the corrector. We show that the corrector can be constructed under either s∈(2,3] or s>3, though the corresponding underlying measures may be different. As an application of the main result we get a new lower bound of the quenched diagonal transition probability for the random walk.
Discrepancy Algorithms for the Binary Perceptron

Shuangping Li,Tselil Schramm,Kangjie Zhou

2024-07-19

Abstract:The binary perceptron problem asks us to find a sign vector in the intersection of independently chosen random halfspaces with intercept $-\kappa$. We analyze the performance of the canonical discrepancy minimization algorithms of Lovett-Meka and Rothvoss/Eldan-Singh for the asymmetric binary perceptron problem. We obtain new algorithmic results in the $\kappa = 0$ case and in the large-$|\kappa|$ case. In the $\kappa\to-\infty$ case, we additionally characterize the storage capacity and complement our algorithmic results with an almost-matching overlap-gap lower bound.

Data Structures and Algorithms,Computational Complexity,Mathematical Physics,Probability
The maximum of a symmetric next neighbor walk on the non-negative integers

Jerome K. Percus,Ora E. Percus

DOI: https://doi.org/10.48550/arXiv.1305.5568

2013-05-24

Abstract:We consider a one-dimensional discrete symmetric random walk with a reflecting boundary at the origin. Generating functions are found for the 2- dimensional probability distribution P{Sn = x,max1?j?n Sn = a} of being at position x after n steps, while the maximal location that the walker has achieved during these n steps is a. We also obtain the familiar (marginal) 1-dimensional distribution for Sn = x, but more importantly that for max1?j?n Sj = a asymptotically at fixed a2/n. We are able to compute and compare the expectations and variances of the two one-dimensional distributions, finding that they have qualitatively similar forms, but differ quantitatively in the anticipated fashion.

Probability
A sharper threshold for bootstrap percolation in two dimensions

Janko Gravner,Alexander E. Holroyd,Robert Morris

DOI: https://doi.org/10.1007/s00440-010-0338-z

IF: 1.944

2010-12-31

Probability Theory and Related Fields

Abstract:Two-dimensional bootstrap percolation is a cellular automaton in which sites become ‘infected’ by contact with two or more already infected nearest neighbours. We consider these dynamics, which can be interpreted as a monotone version of the Ising model, on an n × n square, with sites initially infected independently with probability p. The critical probability pc is the smallest p for which the probability that the entire square is eventually infected exceeds 1/2. Holroyd determined the sharp first-order approximation: pc ~ π2/(18 log n) as n → ∞. Here we sharpen this result, proving that the second term in the expansion is −(log n)−3/2+o(1), and moreover determining it up to a poly(log log n)-factor. The exponent −3/2 corrects numerical predictions from the physics literature.

statistics & probability
Non-Stochastic CDF Estimation Using Threshold Queries

Princewill Okoroafor,Vaishnavi Gupta,Robert Kleinberg,Eleanor Goh

DOI: https://doi.org/10.48550/arXiv.2301.05682

2023-01-14

Abstract:Estimating the empirical distribution of a scalar-valued data set is a basic and fundamental task. In this paper, we tackle the problem of estimating an empirical distribution in a setting with two challenging features. First, the algorithm does not directly observe the data; instead, it only asks a limited number of threshold queries about each sample. Second, the data are not assumed to be independent and identically distributed; instead, we allow for an arbitrary process generating the samples, including an adaptive adversary. These considerations are relevant, for example, when modeling a seller experimenting with posted prices to estimate the distribution of consumers' willingness to pay for a product: offering a price and observing a consumer's purchase decision is equivalent to asking a single threshold query about their value, and the distribution of consumers' values may be non-stationary over time, as early adopters may differ markedly from late adopters. Our main result quantifies, to within a constant factor, the sample complexity of estimating the empirical CDF of a sequence of elements of $[n]$, up to $\varepsilon$ additive error, using one threshold query per sample. The complexity depends only logarithmically on $n$, and our result can be interpreted as extending the existing logarithmic-complexity results for noisy binary search to the more challenging setting where noise is non-stochastic. Along the way to designing our algorithm, we consider a more general model in which the algorithm is allowed to make a limited number of simultaneous threshold queries on each sample. We solve this problem using Blackwell's Approachability Theorem and the exponential weights method. As a side result of independent interest, we characterize the minimum number of simultaneous threshold queries required by deterministic CDF estimation algorithms.

Machine Learning,Data Structures and Algorithms,Econometrics
Understanding the computational difficulty of a binary-weight perceptron and the advantage of input sparseness

Zedong Bi,Changsong Zhou

DOI: https://doi.org/10.1088/1751-8121/ab2682

2019-12-23

Abstract:Limited precision of synaptic weights is a key aspect of both biological and hardware implementationof neural networks. To assign low-precise weights during learning is a non-trivial task, but maybenefit from representing to-be-learned items using sparse code. However, the computationaldifficulty resulting from low weight precision and the advantage of sparse coding remain not fullyunderstood. Here, we study a perceptron model, which associates binary (0 or 1) input patterns withdesired outputs using binary (0 or 1) weights, modeling a single neuron receiving excitatory inputs.Two efficient perceptron solvers (SBPI and rBP) usually find solutions in the dense solution region.We studied this dense-region-approaching process through a decimation algorithm, where at every timestep, marginal probabilities of unfixed weights were evaluated, then the most polarized weight wasfixed at its preferred value. We compared the decimation-fixing order of weights with the dynamicsof ...
Average-case matrix discrepancy: satisfiability bounds

Antoine Maillard

2024-10-23

Abstract:Given a sequence of $d \times d$ symmetric matrices $\{\mathbf{W}_i\}_{i=1}^n$, and a margin $\Delta > 0$, we investigate whether it is possible to find signs $(\epsilon_1, \dots, \epsilon_n) \in \{\pm 1\}^n$ such that the operator norm of the signed sum satisfies $\|\sum_{i=1}^n \epsilon_i \mathbf{W}_i\|_\mathrm{op} \leq \Delta$. Kunisky and Zhang (2023) recently introduced a random version of this problem, where the matrices $\{\mathbf{W}_i\}_{i=1}^n$ are drawn from the Gaussian orthogonal ensemble. This model can be seen as a random variant of the celebrated Matrix Spencer conjecture and as a matrix-valued analog of the symmetric binary perceptron in statistical physics. In this work, we establish a satisfiability transition in this problem as $n, d \to \infty$ with $n / d^2 \to \tau > 0$. Our contributions are twofold. First, we prove that the expected number of solutions with margin $\Delta=\kappa \sqrt{n}$ has a sharp threshold at a critical $\tau_1(\kappa)$: for $\tau < \tau_1(\kappa)$ the problem is typically unsatisfiable, while for $\tau > \tau_1(\kappa)$ the average number of solutions becomes exponentially large. Second, combining a second-moment method with recent results from Altschuler (2023) on margin concentration in perceptron-type problems, we identify a second threshold $\tau_2(\kappa)$, such that for $\tau>\tau_2(\kappa)$ the problem admits solutions with high probability. In particular, we establish for the first time that a system of $n = \Theta(d^2)$ Gaussian random matrices can be balanced so that the spectrum of the resulting matrix macroscopically shrinks compared to the typical semicircle law. Our proofs rely on deriving concentration inequalities and large deviation properties for the law of correlated Gaussian matrices under spectral norm constraints, extending results in the literature and offering insights of independent interest.

Probability,Disordered Systems and Neural Networks,Discrete Mathematics,Combinatorics
A note on the capacity of the binary perceptron

Dylan J. Altschuler,Konstantin Tikhomirov

2024-01-23

Abstract:Determining the capacity $\alpha_c$ of the Binary Perceptron is a long-standing problem. Krauth and Mezard (1989) conjectured an explicit value of $\alpha_c$, approximately equal to .833, and a rigorous lower bound matching this prediction was recently established by Ding and Sun (2019). Regarding the upper bound, Kim and Roche (1998) and Talagrand (1999) independently showed that $\alpha_c$ < .996, while Krauth and Mezard outlined an argument which can be used to show that $\alpha_c$ < .847. The purpose of this expository note is to record a complete proof of the bound $\alpha_c$ < .847. The proof is a conditional first moment method combined with known results on the spherical perceptron

Probability,Discrete Mathematics,Machine Learning
Limit Distributions of Products of Independent and Identically Distributed Random 2 × 2 Stochastic Matrices: A Treatment with the Reciprocal of the Golden Ratio

Santanu Chakraborty

DOI: https://doi.org/10.3390/math11244993

IF: 2.4

2023-12-19

Mathematics

Abstract:Consider a sequence (Xn)n≥1 of i.i.d. 2×2 stochastic matrices with each Xn distributed as μ. This μ is described as follows. Let (Cn,Dn)T denote the first column of Xn and for a given real r with 0<r<1, 0<p1,p2<1.="" 0 12 is very challenging. Considering the extreme nontriviality of this case, we stick to a very special such r, namely, r=5−12 (the reciprocal of the golden ratio), briefly mention the challenges in this nontrivial case, and completely identify λ for a very special situation.</r

mathematics
Clustering of solutions in the symmetric binary perceptron

Carlo Baldassi,Riccardo Della Vecchia,Carlo Lucibello,Riccardo Zecchina

DOI: https://doi.org/10.1088/1742-5468/ab99be

2020-05-12

Abstract:The geometrical features of the (non-convex) loss landscape of neural network models are crucial in ensuring successful optimization and, most importantly, the capability to generalize well. While minimizers' flatness consistently correlates with good generalization, there has been little rigorous work in exploring the condition of existence of such minimizers, even in toy models. Here we consider a simple neural network model, the symmetric perceptron, with binary weights. Phrasing the learning problem as a constraint satisfaction problem, the analogous of a flat minimizer becomes a large and dense cluster of solutions, while the narrowest minimizers are isolated solutions. We perform the first steps toward the rigorous proof of the existence of a dense cluster in certain regimes of the parameters, by computing the first and second moment upper bounds for the existence of pairs of arbitrarily close solutions. Moreover, we present a non rigorous derivation of the same bounds for sets of $y$ solutions at fixed pairwise distances.

Machine Learning
A sharp threshold for a modified bootstrap percolation with recovery

Tom Coker,Karen Gunderson

DOI: https://doi.org/10.48550/arXiv.1505.08030

2015-05-29

Probability

Abstract:Bootstrap percolation is a type of cellular automaton on graphs, introduced as a simple model of the dynamics of ferromagnetism. Vertices in a graph can be in one of two states: `healthy' or `infected' and from an initial configuration of states, healthy vertices become infected by local rules. While the usual bootstrap processes are monotone in the sets of infected vertices, in this paper, a modification is examined in which infected vertices can return to a healthy state. Vertices are initially infected independently at random and the central question is whether all vertices eventually become infected. The model examined here is such a process on a square grid for which healthy vertices with at least two infected neighbours become infected and infected vertices with no infected neighbours become healthy. Sharp thresholds are given for the critical probability of initial infections for all vertices eventually to become infected.
Storage capacity in symmetric binary perceptrons

Benjamin Aubin,Will Perkins,Lenka Zdeborová

DOI: https://doi.org/10.1088/1751-8121/ab227a

2019-06-24

Abstract:We study the problem of determining the capacity of the binary perceptron for two variants of theproblem where the corresponding constraint is symmetric. We call these variants therectangle-binary-perceptron (RPB) and the u -function-binary-perceptron (UBP). We show that, unlikefor the usual step-function-binary-perceptron, the critical capacity in these symmetric cases isgiven by the annealed computation in a large region of parameter space (for all rectangularconstraints and for narrow enough u -function constraints, K
On the Seidel spectrum of threshold graphs

Santanu Mandal,Ranjit Mehatari

DOI: https://doi.org/10.48550/arXiv.2101.03364

2021-01-09

Abstract:In this paper, we analyse spectral properties of Seidel matrix (denoted by $S$) of connected threshold graphs. We compute the characteristic polynomial and determinant of Seidel matrix of threshold graphs. We derive formulas for the multiplicity of the eigenvalues $\pm 1$ of $S$. Further we determine threshold graphs with at most 5 distinct Seidel eigenvalues. Finally we construct families of Seidel cospectral threshold graphs.

Combinatorics,Discrete Mathematics
Sharp Information-Theoretic Thresholds for Shuffled Linear Regression

Leon Lufkin,Yihong Wu,Jiaming Xu

2024-02-15

Abstract:This paper studies the problem of shuffled linear regression, where the correspondence between predictors and responses in a linear model is obfuscated by a latent permutation. Specifically, we consider the model $y = \Pi_* X \beta_* + w$, where $X$ is an $n \times d$ standard Gaussian design matrix, $w$ is Gaussian noise with entrywise variance $\sigma^2$, $\Pi_*$ is an unknown $n \times n$ permutation matrix, and $\beta_*$ is the regression coefficient, also unknown. Previous work has shown that, in the large $n$-limit, the minimal signal-to-noise ratio ($\mathsf{SNR}$), $\lVert \beta_* \rVert^2/\sigma^2$, for recovering the unknown permutation exactly with high probability is between $n^2$ and $n^C$ for some absolute constant $C$ and the sharp threshold is unknown even for $d=1$. We show that this threshold is precisely $\mathsf{SNR} = n^4$ for exact recovery throughout the sublinear regime $d=o(n)$. As a by-product of our analysis, we also determine the sharp threshold of almost exact recovery to be $\mathsf{SNR} = n^2$, where all but a vanishing fraction of the permutation is reconstructed.

Statistics Theory
Distribution of the Ratio of Consecutive Level Spacings for Different Symmetries and Degrees of Chaos

A. L. Corps,A. Relaño

DOI: https://doi.org/10.1103/PhysRevE.101.022222

2020-02-16

Abstract:Theoretical expressions for the distribution of the ratio of consecutive level spacings for quantum systems with transiting dynamics remain unknown. We propose a family of one-parameter distributions $P(r)\equiv P(r;\beta)$, where $\beta\in[0,+\infty)$ is a generalized Dyson index, that describes the eigenlevel statistics of a quantum system characterized by different symmetries and degrees of chaos. We show that this crossover strongly depends on the specific properties of each model, and thus the reduction of such a family to a universal formula, albeit desirable, is not possible. We use the information entropy as a criterion to suggest particular ansatzs for different transitions, with a negligible associated error in the limits corresponding to standard random ensembles.

Quantum Physics,Statistical Mechanics,Mathematical Physics,Chaotic Dynamics

Distribution of the threshold for the symmetric perceptron

Critical Window of The Symmetric Perceptron

On the atypical solutions of the symmetric binary perceptron

Capacity Lower Bound for the Ising Perceptron.

How to escape atypical regions in the symmetric binary perceptron: a journey through connected-solutions states

Scaling Limits for One-Dimensional Long-Range Percolation: Using the Corrector Method

Discrepancy Algorithms for the Binary Perceptron

The maximum of a symmetric next neighbor walk on the non-negative integers

A sharper threshold for bootstrap percolation in two dimensions

Non-Stochastic CDF Estimation Using Threshold Queries

Understanding the computational difficulty of a binary-weight perceptron and the advantage of input sparseness

Average-case matrix discrepancy: satisfiability bounds

A note on the capacity of the binary perceptron

Limit Distributions of Products of Independent and Identically Distributed Random 2 × 2 Stochastic Matrices: A Treatment with the Reciprocal of the Golden Ratio

Clustering of solutions in the symmetric binary perceptron

A sharp threshold for a modified bootstrap percolation with recovery

Storage capacity in symmetric binary perceptrons

On the Seidel spectrum of threshold graphs

Sharp Information-Theoretic Thresholds for Shuffled Linear Regression

Distribution of the Ratio of Consecutive Level Spacings for Different Symmetries and Degrees of Chaos