Perceptron Mistake Bounds

Mehryar Mohri,Afshin Rostamizadeh

DOI: https://doi.org/10.48550/arXiv.1305.0208

IF: 5.414

2013-05-01

Machine Learning

Abstract:We present a brief survey of existing mistake bounds and introduce novel bounds for the Perceptron or the kernel Perceptron algorithm. Our novel bounds generalize beyond standard margin-loss type bounds, allow for any convex and Lipschitz loss function, and admit a very simple proof.

What problem does this paper attempt to address?

The Perceptron with Dynamic Margin

Constantinos Panagiotakopoulos,Petroula Tsampouka

DOI: https://doi.org/10.48550/arXiv.1105.6041

IF: 5.414

2011-05-30

Machine Learning

Abstract:The classical perceptron rule provides a varying upper bound on the maximum margin, namely the length of the current weight vector divided by the total number of updates up to that time. Requiring that the perceptron updates its internal state whenever the normalized margin of a pattern is found not to exceed a certain fraction of this dynamic upper bound we construct a new approximate maximum margin classifier called the perceptron with dynamic margin (PDM). We demonstrate that PDM converges in a finite number of steps and derive an upper bound on them. We also compare experimentally PDM with other perceptron-like algorithms and support vector machines on hard margin tasks involving linear kernels which are equivalent to 2-norm soft margin.
Discrepancy Algorithms for the Binary Perceptron

Shuangping Li,Tselil Schramm,Kangjie Zhou

2024-07-19

Abstract:The binary perceptron problem asks us to find a sign vector in the intersection of independently chosen random halfspaces with intercept $-\kappa$. We analyze the performance of the canonical discrepancy minimization algorithms of Lovett-Meka and Rothvoss/Eldan-Singh for the asymmetric binary perceptron problem. We obtain new algorithmic results in the $\kappa = 0$ case and in the large-$|\kappa|$ case. In the $\kappa\to-\infty$ case, we additionally characterize the storage capacity and complement our algorithmic results with an almost-matching overlap-gap lower bound.

Data Structures and Algorithms,Computational Complexity,Mathematical Physics,Probability
On the Perceptron's Compression

Shay Moran,Ido Nachum,Itai Panasoff,Amir Yehudayoff

DOI: https://doi.org/10.48550/arXiv.1806.05403

IF: 5.414

2018-06-14

Machine Learning

Abstract:We study and provide exposition to several phenomena that are related to the perceptron's compression. One theme concerns modifications of the perceptron algorithm that yield better guarantees on the margin of the hyperplane it outputs. These modifications can be useful in training neural networks as well, and we demonstrate them with some experimental data. In a second theme, we deduce conclusions from the perceptron's compression in various contexts.
Ahpatron: A New Budgeted Online Kernel Learning Machine with Tighter Mistake Bound

Yun Liao,Junfan Li,Shizhong Liao,Qinghua Hu,Jianwu Dang

2023-12-12

Abstract:In this paper, we study the mistake bound of online kernel learning on a budget. We propose a new budgeted online kernel learning model, called Ahpatron, which significantly improves the mistake bound of previous work and resolves the open problem posed by Dekel, Shalev-Shwartz, and Singer (2005). We first present an aggressive variant of Perceptron, named AVP, a model without budget, which uses an active updating rule. Then we design a new budget maintenance mechanism, which removes a half of examples,and projects the removed examples onto a hypothesis space spanned by the remaining examples. Ahpatron adopts the above mechanism to approximate AVP. Theoretical analyses prove that Ahpatron has tighter mistake bounds, and experimental results show that Ahpatron outperforms the state-of-the-art algorithms on the same or a smaller budget.

Machine Learning
On-line learning through simple perceptron with a margin

Kazuyuki Hara,Masato Okada

DOI: https://doi.org/10.48550/arXiv.cond-mat/0306150

2003-06-05

Disordered Systems and Neural Networks

Abstract:We analyze a learning method that uses a margin $\kappa$ {\it a la} Gardner for simple perceptron learning. This method corresponds to the perceptron learning when $\kappa=0$, and to the Hebbian learning when $\kappa \to \infty$. Nevertheless, we found that the generalization ability of the method was superior to that of the perceptron and the Hebbian methods at an early stage of learning. We analyzed the asymptotic property of the learning curve of this method through computer simulation and found that it was the same as for perceptron learning. We also investigated an adaptive margin control method.
Generalization Errors of the Simple Perceptron

JF Feng

DOI: https://doi.org/10.1088/0305-4470/31/17/013

1998-01-01

Abstract:To find an exact form for the generalization error of a learning machine is an open problem, even in the simplest case: simple perceptron learning. We introduce a new approach to tackle the problem. The generalization error of the simple perceptron is expressed as a linear combination of extreme values of inputs. With the help of extreme value theory in statistics we then obtain an exact form of the generalization error of the simple perceptron in the case of the worst learning. Generalization errors of the higher-order perceptron taking the form of an inverse power law in the number of examples are also considered.
On Accelerated Perceptrons and Beyond

Guanghui Wang,Rafael Hanashiro,Etash Guha,Jacob Abernethy

DOI: https://doi.org/10.48550/arXiv.2210.09371

2022-10-18

Abstract:The classical Perceptron algorithm of Rosenblatt can be used to find a linear threshold function to correctly classify $n$ linearly separable data points, assuming the classes are separated by some margin $\gamma > 0$. A foundational result is that Perceptron converges after $\Omega(1/\gamma^{2})$ iterations. There have been several recent works that managed to improve this rate by a quadratic factor, to $\Omega(\sqrt{\log n}/\gamma)$, with more sophisticated algorithms. In this paper, we unify these existing results under one framework by showing that they can all be described through the lens of solving min-max problems using modern acceleration techniques, mainly through optimistic online learning. We then show that the proposed framework also lead to improved results for a series of problems beyond the standard Perceptron setting. Specifically, a) For the margin maximization problem, we improve the state-of-the-art result from $O(\log t/t^2)$ to $O(1/t^2)$, where $t$ is the number of iterations; b) We provide the first result on identifying the implicit bias property of the classical Nesterov's accelerated gradient descent (NAG) algorithm, and show NAG can maximize the margin with an $O(1/t^2)$ rate; c) For the classical $p$-norm Perceptron problem, we provide an algorithm with $\Omega(\sqrt{(p-1)\log n}/\gamma)$ convergence rate, while existing algorithms suffer the $\Omega({(p-1)}/\gamma^2)$ convergence rate.

Machine Learning,Optimization and Control
A note on the capacity of the binary perceptron

Dylan J. Altschuler,Konstantin Tikhomirov

2024-01-23

Abstract:Determining the capacity $\alpha_c$ of the Binary Perceptron is a long-standing problem. Krauth and Mezard (1989) conjectured an explicit value of $\alpha_c$, approximately equal to .833, and a rigorous lower bound matching this prediction was recently established by Ding and Sun (2019). Regarding the upper bound, Kim and Roche (1998) and Talagrand (1999) independently showed that $\alpha_c$ < .996, while Krauth and Mezard outlined an argument which can be used to show that $\alpha_c$ < .847. The purpose of this expository note is to record a complete proof of the bound $\alpha_c$ < .847. The proof is a conditional first moment method combined with known results on the spherical perceptron

Probability,Discrete Mathematics,Machine Learning
Mistake, Manipulation and Margin Guarantees in Online Strategic Classification

Lingqing Shen,Nam Ho-Nguyen,Khanh-Hung Giang-Tran,Fatma Kılınç-Karzan

2024-03-27

Abstract:We consider an online strategic classification problem where each arriving agent can manipulate their true feature vector to obtain a positive predicted label, while incurring a cost that depends on the amount of manipulation. The learner seeks to predict the agent's true label given access to only the manipulated features. After the learner releases their prediction, the agent's true label is revealed. Previous algorithms such as the strategic perceptron guarantee finitely many mistakes under a margin assumption on agents' true feature vectors. However, these are not guaranteed to encourage agents to be truthful. Promoting truthfulness is intimately linked to obtaining adequate margin on the predictions, thus we provide two new algorithms aimed at recovering the maximum margin classifier in the presence of strategic agent behavior. We prove convergence, finite mistake and finite manipulation guarantees for a variety of agent cost structures. We also provide generalized versions of the strategic perceptron with mistake guarantees for different costs. Our numerical study on real and synthetic data demonstrates that the new algorithms outperform previous ones in terms of margin, number of manipulation and number of mistakes.

Machine Learning,Computer Science and Game Theory,Optimization and Control
On the Atypical Solutions of the Symmetric Binary Perceptron

Damien Barbier,Ahmed El Alaoui,Florent Krzakala,Lenka Zdeborová

2024-06-28

Abstract:We study the random binary symmetric perceptron problem, focusing on the behavior of rare high-margin solutions. While most solutions are isolated, we demonstrate that these rare solutions are part of clusters of extensive entropy, heuristically corresponding to non-trivial fixed points of an approximate message-passing algorithm. We enumerate these clusters via a local entropy, defined as a Franz-Parisi potential, which we rigorously evaluate using the first and second moment methods in the limit of a small constraint density $\alpha$ (corresponding to vanishing margin $\kappa$) under a certain assumption on the concentration of the entropy. This examination unveils several intriguing phenomena: i) We demonstrate that these clusters have an entropic barrier in the sense that the entropy as a function of the distance from the reference high-margin solution is non-monotone when $\kappa \le 1.429 \sqrt{-\alpha/\log{\alpha}}$, while it is monotone otherwise, and that they have an energetic barrier in the sense that there are no solutions at an intermediate distance from the reference solution when $\kappa \le 1.239 \sqrt{-\alpha/ \log{\alpha}}$. The critical scaling of the margin $\kappa$ in $\sqrt{-\alpha/\log\alpha}$ corresponds to the one obtained from the earlier work of Gamarnik et al. (2022) for the overlap-gap property, a phenomenon known to present a barrier to certain efficient algorithms. ii) We establish using the replica method that the complexity (the logarithm of the number of clusters of such solutions) versus entropy (the logarithm of the number of solutions in the clusters) curves are partly non-concave and correspond to very large values of the Parisi parameter, with the equilibrium being reached when the Parisi parameter diverges.

Probability,Disordered Systems and Neural Networks
Quantum Perceptron Revisited: Computational-Statistical Tradeoffs

Mathieu Roget,Giuseppe Di Molfetta,Hachem Kadri

DOI: https://doi.org/10.48550/arXiv.2106.02496

2021-06-04

Quantum Physics

Abstract:Quantum machine learning algorithms could provide significant speed-ups over their classical counterparts; however, whether they could also achieve good generalization remains unclear. Recently, two quantum perceptron models which give a quadratic improvement over the classical perceptron algorithm using Grover's search have been proposed by Wiebe et al. arXiv:1602.04799 . While the first model reduces the complexity with respect to the size of the training set, the second one improves the bound on the number of mistakes made by the perceptron. In this paper, we introduce a hybrid quantum-classical perceptron algorithm with lower complexity and better generalization ability than the classical perceptron. We show a quadratic improvement over the classical perceptron in both the number of samples and the margin of the data. We derive a bound on the expected error of the hypothesis returned by our algorithm, which compares favorably to the one obtained with the classical online perceptron. We use numerical experiments to illustrate the trade-off between computational complexity and statistical accuracy in quantum perceptron learning and discuss some of the key practical issues surrounding the implementation of quantum perceptron models into near-term quantum devices, whose practical implementation represents a serious challenge due to inherent noise. However, the potential benefits make correcting this worthwhile.
Estimating Exact Form of Generalisation Errors

Jianfeng Feng

DOI: https://doi.org/10.1007/bfb0098198

1999-01-01

Abstract:A novel approach to estimate generalisation errors of the simple perceptron of the worst case is introduced. It is well known that the generalisation error of the simple perceptron is of the form d/t with an unknown constant d which depends only on the dimension of inputs, where t is the number of learned examples. Based upon extreme value theory in statistics we obtain an exact form of the generalisation error of the simple perceptron. The method introduced in this paper opens up new possibilities to consider generalisation errors of a class of neural networks.
Error Bounds for a Kernel-Based Constrained Optimal Smoothing Approximation

Laurence Grammont,François Bachoc,Andrés F. López-Lopera

2024-07-12

Abstract:This paper establishes error bounds for the convergence of a piecewise linear approximation of the constrained optimal smoothing problem posed in a reproducing kernel Hilbert space (RKHS). This problem can be reformulated as a Bayesian estimation problem involving a Gaussian process related to the kernel of the RKHS. Consequently, error bounds can be interpreted as a quantification of the maximum a posteriori (MAP) accuracy. To our knowledge, no error bounds have been proposed for this type of problem so far. The convergence results are provided as a function of the grid size, the regularity of the kernel, and the distance from the kernel interpolant of the approximation to the set of constraints. Inspired by the MaxMod algorithm from recent literature, which sequentially allocates knots for the piecewise linear approximation, we conduct our analysis for non-equispaced knots. These knots are even allowed to be non-dense, which impacts the definition of the optimal smoothing solution and our error bound quantifiers. Finally, we illustrate our theorems through several numerical experiments involving constraints such as boundedness and monotonicity.

Statistics Theory
Improved Error Bounds Based on Worst Likely Assignments

Eric Bax

DOI: https://doi.org/10.48550/arXiv.1504.00052

2015-04-01

Abstract:Error bounds based on worst likely assignments use permutation tests to validate classifiers. Worst likely assignments can produce effective bounds even for data sets with 100 or fewer training examples. This paper introduces a statistic for use in the permutation tests of worst likely assignments that improves error bounds, especially for accurate classifiers, which are typically the classifiers of interest.

Machine Learning,Information Theory,Probability
Tractability from overparametrization: the example of the negative perceptron

Andrea Montanari,Yiqiao Zhong,Kangjie Zhou

DOI: https://doi.org/10.1007/s00440-023-01248-y

IF: 1.944

2024-01-24

Probability Theory and Related Fields

Abstract:In the negative perceptron problem we are given n data points , where is a d -dimensional vector and is a binary label. The data are not linearly separable and hence we content ourselves to find a linear classifier with the largest possible negative margin. In other words, we want to find a unit norm vector that maximizes . This is a non-convex optimization problem (it is equivalent to finding a maximum norm vector in a polytope), and we study its typical properties under two random models for the data. We consider the proportional asymptotics in which with , and prove upper and lower bounds on the maximum margin or—equivalently—on its inverse function . In other words, is the overparametrization threshold: for a classifier achieving vanishing training error exists with high probability, while for it does not. Our bounds on match to the leading order as . We then analyze a linear programming algorithm to find a solution, and characterize the corresponding threshold . We observe a gap between the interpolation threshold and the linear programming threshold , raising the question of the behavior of other algorithms.

statistics & probability
Critical Window of The Symmetric Perceptron

Dylan J. Altschuler

2023-08-09

Abstract:We study the critical window of the symmetric binary perceptron, or equivalently, combinatorial discrepancy. Consider the problem of finding a binary vector $\sigma$ satisfying $\|A\sigma\|_\infty \le K$, where $A$ is an $\alpha n \times n$ matrix with iid Gaussian entries. For fixed $K$, at which densities $\alpha$ is this constraint satisfaction problem (CSP) satisfiable? A sharp threshold was recently established by Perkins and Xu, and Abbe, Li, and Sly , answering this to first order. Namely, for each $K$ there exists an explicit critical density $\alpha_c$ so that for any fixed $\epsilon > 0$, with high probability the CSP is satisfiable for $\alpha n < (\alpha_c - \epsilon ) n$ and unsatisfiable for $\alpha n > (\alpha_c + \epsilon) n$. This corresponds to a bound of $o(n)$ on the size of the critical window. We sharpen these results significantly, as well as provide exponential tail bounds. Our main result is that, perhaps surprisingly, the critical window is actually at most $O(\log n)$. More precisely, with high probability the CSP is satisfiable for $\alpha n < \alpha_c n -O(\log n)$ and unsatisfiable for any $\alpha n > \alpha_c n + \omega(1)$. This implies the symmetric perceptron has nearly the "sharpest possible transition," adding it to a short list of CSP for which the critical window is rigorously known to be of near-constant width.

Probability,Discrete Mathematics,Mathematical Physics,Combinatorics
Sharper Bounds for Proximal Gradient Algorithms with Errors

Anis Hamadouche,Yun Wu,Andrew M. Wallace,João F. C. Mota

DOI: https://doi.org/10.1137/22m1480161

IF: 2.763

2024-01-20

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 1, Page 278-305, March 2024. We analyze the convergence of the proximal gradient algorithm for convex composite problems in the presence of gradient and proximal computational inaccuracies. We generalize the deterministic analysis to the quasi-Fejér case and quantify the uncertainty incurred from approximate computing and early termination errors. We propose new probabilistic tighter bounds that we use to verify a simulated Model Predictive Control (MPC) with sparse controls problem solved with early termination, reduced precision, and proximal errors. We also show how the probabilistic bounds are more suitable than the deterministic ones for algorithm verification and more accurate for application performance guarantees. Under mild statistical assumptions, we also prove that some cumulative error terms follow a martingale property. And conforming to observations, e.g., in [M. Schmidt, N. L. Roux, and F. R. Bach, Convergence rates of inexact proximal-gradient methods for convex optimization, in Advances in Neural Information Processing Systems, 2011, pp. 1458–1466], we also show how the acceleration of the algorithm amplifies the gradient and proximal computational errors.

mathematics, applied
Generalized Power Cones: Optimal Error Bounds and Automorphisms

Ying Lin,Scott B. Lindstrom,Bruno F. Lourenço,Ting Kei Pong

DOI: https://doi.org/10.1137/22m1542921

IF: 2.763

2024-04-10

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 2, Page 1316-1340, June 2024. Error bounds are a requisite for trusting or distrusting solutions in an informed way. Until recently, provable error bounds in the absence of constraint qualifications were unattainable for many classes of cones that do not admit projections with known succinct expressions. We build such error bounds for the generalized power cones, using the recently developed framework of one-step facial residual functions. We also show that our error bounds are tight in the sense of that framework. Besides their utility for understanding solution reliability, the error bounds we discover have additional applications to the algebraic structure of the underlying cone, which we describe. In particular we use the error bounds to compute the automorphisms of the generalized power cones, and to identify a set of generalized power cones that are self-dual, irreducible, nonhomogeneous, and perfect.

mathematics, applied
Capacity Lower Bound for the Ising Perceptron.

Jian Ding,Nike Sun

DOI: https://doi.org/10.1145/3313276.3316383

2019-01-01

Abstract:We consider the Ising perceptron with gaussian disorder, which is equivalent to the discrete cube {−1,+1}N intersected by M random half-spaces. The perceptron’s capacity is the largest integer MN for which the intersection is nonempty. It is conjectured by Krauth and Mézard (1989) that the (random) ratio MN/N converges in probability to an explicit constant α⋆≐ 0.83. Kim and Roche (1998) proved the existence of a positive constant γ such that γ ≤ MN/N ≤ 1−γ with high probability; see also Talagrand (1999). In this paper we show that the Krauth–Mézard conjecture α⋆ is a lower bound with positive probability, under the condition that an explicit univariate function S(λ) is maximized at λ=0. Our proof is an application of the second moment method to a certain slice of perceptron configurations, as selected by the so-called TAP (Thouless, Anderson, and Palmer, 1977) or AMP (approximate message passing) iteration, whose scaling limit has been characterized by Bayati and Montanari (2011) and Bolthausen (2012). For verifying the condition on S(λ) we outline one approach, which is implemented in the current version using (nonrigorous) numerical integration packages. In a future version of this paper we intend to complete the verification by implementing a rigorous numerical method.

Perceptron Mistake Bounds

The Perceptron with Dynamic Margin

Discrepancy Algorithms for the Binary Perceptron

On the Perceptron's Compression

Ahpatron: A New Budgeted Online Kernel Learning Machine with Tighter Mistake Bound

On-line learning through simple perceptron with a margin

Generalization Errors of the Simple Perceptron

On Accelerated Perceptrons and Beyond

A note on the capacity of the binary perceptron

Mistake, Manipulation and Margin Guarantees in Online Strategic Classification

On the Atypical Solutions of the Symmetric Binary Perceptron

Quantum Perceptron Revisited: Computational-Statistical Tradeoffs

Estimating Exact Form of Generalisation Errors

Error Bounds for a Kernel-Based Constrained Optimal Smoothing Approximation

Improved Error Bounds Based on Worst Likely Assignments

Tractability from overparametrization: the example of the negative perceptron

Critical Window of The Symmetric Perceptron

Sharper Bounds for Proximal Gradient Algorithms with Errors

Generalized Power Cones: Optimal Error Bounds and Automorphisms

Capacity Lower Bound for the Ising Perceptron.