Learning Reductions that Really Work

Alina Beygelzimer,Hal Daumé III,John Langford,Paul Mineiro

DOI: https://doi.org/10.48550/arXiv.1502.02704

2015-02-10

Abstract:We provide a summary of the mathematical and computational techniques that have enabled learning reductions to effectively address a wide class of problems, and show that this approach to solving machine learning problems can be broadly useful.

Machine Learning

What problem does this paper attempt to address?

Tool-Assisted Learning of Computational Reductions

Tristan Kneisel,Elias Radtke,Marko Schmellenkamp,Fabian Vehlken,Thomas Zeume

2024-10-05

Abstract:Computational reductions are an important and powerful concept in computer science. However, they are difficult for many students to grasp. In this paper, we outline a concept for how the learning of reductions can be supported by educational support systems. We present an implementation of the concept within such a system, concrete web-based and interactive learning material for reductions, and report on our experiences using the material in a large introductory course on theoretical computer science.

Computers and Society,Human-Computer Interaction
Verified reductions for optimization

Alexander Bentkamp,Ramon Fernández Mir,Jeremy Avigad

DOI: https://doi.org/10.48550/arXiv.2301.09347

2023-02-22

Abstract:Numerical and symbolic methods for optimization are used extensively in engineering, industry, and finance. Various methods are used to reduce problems of interest to ones that are amenable to solution by such software. We develop a framework for designing and applying such reductions, using the Lean programming language and interactive proof assistant. Formal verification makes the process more reliable, and the availability of an interactive framework and ambient mathematical library provides a robust environment for constructing the reductions and reasoning about them.

Logic in Computer Science,Optimization and Control
Optimality-based domain reduction for inequality-constrained NLP and MINLP problems

Yi Zhang,Nikolaos V. Sahinidis,Carlos Nohra,Gang Rong

DOI: https://doi.org/10.1007/s10898-020-00886-z

2020-01-01

Journal of Global Optimization

Abstract:In spatial branch-and-bound algorithms, optimality-based domain reduction is normally performed after solving a node and relies on duality information to reduce ranges of variables. In this work, we propose novel optimality conditions for NLP and MINLP problems and apply them for domain reduction prior to solving a node in branch-and-bound. The conditions apply to nonconvex inequality-constrained problems for which we exploit monotonicity properties of objectives and constraints. We develop three separate reduction algorithms for unconstrained, one-constraint, and multi-constraint problems. We use the optimality conditions to reduce ranges of variables through forward and backward bound propagation of gradients respective to each decision variable. We describe an efficient implementation of these techniques in the branch-and-bound solver BARON. The implementation dynamically recognizes and ignores inactive constraints at each node of the search tree. Our computations demonstrate that the proposed techniques often reduce the solution time and total number of nodes for continuous problems; they are less effective for mixed-integer programs.
Knowledge Reduction in Real Decision Formal Contexts

Jinhai Li,Changlin Mei,Yuejin Lv,Xiao Zhang

DOI: https://doi.org/10.1016/j.ins.2011.11.041

IF: 8.1

2011-01-01

Information Sciences

Abstract:Computing a minimal reduct of a decision formal context by Boolean reasoning is an NP-hard problem. Thus, it is essential to develop some heuristic methods to deal with the issue of knowledge reduction especially for large decision formal contexts. In this study, we first investigate the relationship between the concept lattice of a formal context and those of its subcontexts in preparation for deriving a heuristic knowledge-reduction method. Then, we construct a new framework of knowledge reduction in which the capacity of one concept lattice implying another is defined to measure the significance of the attributes in a consistent decision formal context. Based on this reduction framework, we formulate an algorithm of searching for a minimal reduct of a consistent decision formal context. It is proved that this algorithm is complete and its time complexity is polynomial. Some numerical experiments demonstrate that the algorithm can generally obtain a minimal reduct and is much more efficient than some Boolean reasoning-based methods.
The Neural Race Reduction: Dynamics of Abstraction in Gated Networks

Andrew M. Saxe,Shagun Sodhani,Sam Lewallen

DOI: https://doi.org/10.48550/arXiv.2207.10430

2022-07-21

Abstract:Our theoretical understanding of deep learning has not kept pace with its empirical success. While network architecture is known to be critical, we do not yet understand its effect on learned representations and network behavior, or how this architecture should reflect task <a class="link-external link-http" href="http://structure.In" rel="external noopener nofollow">this http URL</a> this work, we begin to address this gap by introducing the Gated Deep Linear Network framework that schematizes how pathways of information flow impact learning dynamics within an architecture. Crucially, because of the gating, these networks can compute nonlinear functions of their input. We derive an exact reduction and, for certain cases, exact solutions to the dynamics of learning. Our analysis demonstrates that the learning dynamics in structured networks can be conceptualized as a neural race with an implicit bias towards shared representations, which then govern the model's ability to systematically generalize, multi-task, and transfer. We validate our key insights on naturalistic datasets and with relaxed assumptions. Taken together, our work gives rise to general hypotheses relating neural architecture to learning and provides a mathematical approach towards understanding the design of more complex architectures and the role of modularity and compositionality in solving real-world problems. The code and results are available at <a class="link-external link-https" href="https://www.saxelab.org/gated-dln" rel="external noopener nofollow">this https URL</a> .

Machine Learning,Artificial Intelligence
Logical reduction of metarules

Andrew Cropper,Sophie Tourret

DOI: https://doi.org/10.1007/s10994-019-05834-x

IF: 5.414

2019-11-20

Machine Learning

Abstract:Abstract Many forms of inductive logic programming (ILP) use metarules , second-order Horn clauses, to define the structure of learnable programs and thus the hypothesis space. Deciding which metarules to use for a given learning task is a major open problem and is a trade-off between efficiency and expressivity: the hypothesis space grows given more metarules, so we wish to use fewer metarules, but if we use too few metarules then we lose expressivity. In this paper, we study whether fragments of metarules can be logically reduced to minimal finite subsets. We consider two traditional forms of logical reduction: subsumption and entailment. We also consider a new reduction technique called derivation reduction , which is based on SLD-resolution. We compute reduced sets of metarules for fragments relevant to ILP and theoretically show whether these reduced sets are reductions for more general infinite fragments. We experimentally compare learning with reduced sets of metarules on three domains: Michalski trains, string transformations, and game rules. In general, derivation reduced sets of metarules outperform subsumption and entailment reduced sets, both in terms of predictive accuracies and learning times.

computer science, artificial intelligence
Experimental Comparisons of Instances Set Reduction Algorithms

yuelin yu,yangguang liu,bin xu,xiaoqi he

DOI: https://doi.org/10.1007/978-3-642-37829-4_52

2014-01-01

Abstract:As techniques of data acquisition and data storage rapidly developed, more and larger datasets are very easily faced in machine learning. In order to avoid excessive storage and time consuming, and possibly to improve generalization accuracy by removing noise, several works presented as reduction techniques have been proposed. In this paper, firstly, we will review most traditional and typical reduction algorithms and find out their strengths and weaknesses, respectively. In addition, nine typical reduction algorithms are compared performing on 16 classification tasks. At last, some valuable directions for further research are proposed based on discussions and conclusion of traditional algorithms mentioned.
Fully Automatic Neural Network Reduction for Formal Verification

Tobias Ladner,Matthias Althoff

2024-04-23

Abstract:Formal verification of neural networks is essential before their deployment in safety-critical applications. However, existing methods for formally verifying neural networks are not yet scalable enough to handle practical problems involving a large number of neurons. We address this challenge by introducing a fully automatic and sound reduction of neural networks using reachability analysis. The soundness ensures that the verification of the reduced network entails the verification of the original network. To the best of our knowledge, we present the first sound reduction approach that is applicable to neural networks with any type of element-wise activation function, such as ReLU, sigmoid, and tanh. The network reduction is computed on the fly while simultaneously verifying the original network and its specifications. All parameters are automatically tuned to minimize the network size without compromising verifiability. We further show the applicability of our approach to convolutional neural networks by explicitly exploiting similar neighboring pixels. Our evaluation shows that our approach can reduce the number of neurons to a fraction of the original number of neurons with minor outer-approximation and thus reduce the verification time to a similar degree.

Machine Learning
Impact of domain reduction techniques in polynomial optimization: A computational study

Ignacio Gómez-Casares,Brais González-Rodríguez,Julio González-Díaz,Pablo Rodríguez-Fernández

2024-03-05

Abstract:Domain reduction techniques are at the core of any global optimization solver for NLP or MINLP problems. In this paper, we delve into several of these techniques and assess the impact they may have in the performance of an RLT-based algorithm for polynomial optimization problems. These techniques include i) the use of (nonlinear) conic relaxations for optimality-based bound tightening, ii) the use of Lagrangian dual information to enhance feasibility-based bound tightening, and iii) different strategies for branching point selection. We also explore how a solver equipped with these domain reduction enhancements can further improve its performance by using machine learning to better choose the best domain reduction approach to use on a given instance.

Optimization and Control
Learning with Known Operators reduces Maximum Training Error Bounds

Andreas K. Maier,Christopher Syben,Bernhard Stimpel,Tobias Würfl,Mathis Hoffmann,Frank Schebesch,Weilin Fu,Leonid Mill,Lasse Kling,Silke Christiansen

DOI: https://doi.org/10.1038/s42256-019-0077-5

2019-07-03

Abstract:We describe an approach for incorporating prior knowledge into machine learning algorithms. We aim at applications in physics and signal processing in which we know that certain operations must be embedded into the algorithm. Any operation that allows computation of a gradient or sub-gradient towards its inputs is suited for our framework. We derive a maximal error bound for deep nets that demonstrates that inclusion of prior knowledge results in its reduction. Furthermore, we also show experimentally that known operators reduce the number of free parameters. We apply this approach to various tasks ranging from CT image reconstruction over vessel segmentation to the derivation of previously unknown imaging algorithms. As such the concept is widely applicable for many researchers in physics, imaging, and signal processing. We assume that our analysis will support further investigation of known operators in other fields of physics, imaging, and signal processing.

Machine Learning,Computer Vision and Pattern Recognition,Medical Physics
Knowledge Reduction for Formal Context Based on Concept Lattice

Hong WANG,Wen-Xiu ZHANG

DOI: https://doi.org/10.3969/j.issn.1003-6059.2005.06.001

2005-01-01

Abstract:Formal concept analysis has very important meaning in mathematical methods for data mining and knowledge processing. Knowledge reduction in formal context based on concept lattice is dealt with in this paper. By the similar idea in knowledge reduction of information systems in rough set theory, the judgement theorems of knowledge reduction for formal context in concept lattice and discernibility matrices are proposed. Furthermore, the approaches to knowledge reduction in concept lattice are presented.
On the convergence of reduction-based and model-based methods in proof theory

Gilles Dowek

2023-05-02

Abstract:In the recent past, the reduction-based and the model-based methods to prove cut elimination have converged, so that they now appear just as two sides of the same coin. This paper details some of the steps of this transformation.

Logic in Computer Science
LeanAgent: Lifelong Learning for Formal Theorem Proving

Adarsh Kumarappan,Mo Tiwari,Peiyang Song,Robert Joseph George,Chaowei Xiao,Anima Anandkumar

2024-10-30

Abstract:Large Language Models (LLMs) have been successful in mathematical reasoning tasks such as formal theorem proving when integrated with interactive proof assistants like Lean. Existing approaches involve training or fine-tuning an LLM on a specific dataset to perform well on particular domains, such as undergraduate-level mathematics. These methods struggle with generalizability to advanced mathematics. A fundamental limitation is that these approaches operate on static domains, failing to capture how mathematicians often work across multiple domains and projects simultaneously or cyclically. We present LeanAgent, a novel lifelong learning framework for theorem proving that continuously generalizes to and improves on ever-expanding mathematical knowledge without forgetting previously learned knowledge. LeanAgent introduces several key innovations, including a curriculum learning strategy that optimizes the learning trajectory in terms of mathematical difficulty, a dynamic database for efficient management of evolving mathematical knowledge, and progressive training to balance stability and plasticity. LeanAgent successfully proves 162 theorems previously unproved by humans across 23 diverse Lean repositories, many from advanced mathematics. It performs significantly better than the static LLM baseline, proving challenging theorems in domains like abstract algebra and algebraic topology while showcasing a clear progression of learning from basic concepts to advanced topics. In addition, we analyze LeanAgent's superior performance on key lifelong learning metrics. LeanAgent achieves exceptional scores in stability and backward transfer, where learning new tasks improves performance on previously learned tasks. This emphasizes LeanAgent's continuous generalizability and improvement, explaining its superior theorem-proving performance.

Machine Learning,Artificial Intelligence,Logic in Computer Science
Compressing neural networks via formal methods

Dalila Ressi,Riccardo Romanello,Sabina Rossi,Carla Piazza

DOI: https://doi.org/10.1016/j.neunet.2024.106411

IF: 7.8

2024-05-31

Neural Networks

Abstract:Advancements in Neural Networks have led to larger models, challenging implementation on embedded devices with memory, battery, and computational constraints. Consequently, network compression has flourished, offering solutions to reduce operations and parameters. However, many methods rely on heuristics, often requiring re-training for accuracy. Model reduction techniques extend beyond Neural Networks, relevant in Verification and Performance Evaluation fields. This paper bridges widely-used reduction strategies with formal concepts like lumpability, designed for analyzing Markov Chains. We propose a pruning approach based on lumpability, preserving exact behavioral outcomes without data dependence or fine-tuning. Relaxing strict quotienting method definitions enables a formal understanding of common reduction techniques.

computer science, artificial intelligence,neurosciences
Achieving Occam's Razor: Deep Learning for Optimal Model Reduction

Botond B Antal,Anthony G Chesebro,Helmut H Strey,Lilianne R Mujica-Parodi,Corey Weistuch

2023-09-07

Abstract:All fields of science depend on mathematical models. Occam's razor refers to the principle that good models should exclude parameters beyond those minimally required to describe the systems they represent. This is because redundancy can lead to incorrect estimates of model parameters from data, and thus inaccurate or ambiguous conclusions. Here, we show how deep learning can be powerfully leveraged to address Occam's razor. FixFit, our new method, uses a feedforward deep neural network with a bottleneck layer to characterize and predict the behavior of a given model from its input parameters. FixFit has three major benefits. First, it provides a metric to quantify the original model's degree of complexity. Second, it allows for the unique fitting of data. Third, it provides an unbiased way to discriminate between experimental hypotheses that add value versus those that do not. In two use cases, we demonstrate the broad applicability of this method across scientific domains. To validate the method using a known system, we apply FixFit to recover known composite parameters for the Kepler orbit model. To illustrate how the method can be applied to less well-established fields, we use it to identify parameters for a multi-scale brain model and reduce the search space for viable candidate mechanisms.

Machine Learning,Optimization and Control
SiMpLIfy: A Toolbox for Structured Model Reduction

Martin Biel,Farhad Farokhi,Henrik Sandberg

DOI: https://doi.org/10.48550/arXiv.1410.4613

2014-10-17

Abstract:In this paper, we present a toolbox for structured model reduction developed for MATLAB. In addition to structured model reduction methods using balanced realizations of the subsystems, we introduce a numerical algorithm for structured model reduction using a subgradient optimization algorithm. We briefly present the syntax for the toolbox and its features. Finally, we demonstrate the applicability of various model reduction methods in the toolbox on a structured mass-spring mechanical system.

Optimization and Control,Systems and Control
Learning advanced mathematical computations from examples

François Charton,Amaury Hayat,Guillaume Lample

DOI: https://doi.org/10.48550/arXiv.2006.06462

2021-03-19

Abstract:Using transformers over large generated datasets, we train models to learn mathematical properties of differential systems, such as local stability, behavior at infinity and controllability. We achieve near perfect prediction of qualitative characteristics, and good approximations of numerical features of the system. This demonstrates that neural networks can learn to perform complex computations, grounded in advanced theory, from examples, without built-in mathematical knowledge.

Machine Learning,Computation and Language
Machine Learning with Guarantees using Descriptive Complexity and SMT Solvers

Charles Jordan,Łukasz Kaiser

DOI: https://doi.org/10.48550/arXiv.1609.02664

2016-09-09

Abstract:Machine learning is a thriving part of computer science. There are many efficient approaches to machine learning that do not provide strong theoretical guarantees, and a beautiful general learning theory. Unfortunately, machine learning approaches that give strong theoretical guarantees have not been efficient enough to be applicable. In this paper we introduce a logical approach to machine learning. Models are represented by tuples of logical formulas and inputs and outputs are logical structures. We present our framework together with several applications where we evaluate it using SAT and SMT solvers. We argue that this approach to machine learning is particularly suited to bridge the gap between efficiency and theoretical soundness. We exploit results from descriptive complexity theory to prove strong theoretical guarantees for our approach. To show its applicability, we present experimental results including learning complexity-theoretic reductions rules for board games. We also explain how neural networks fit into our framework, although the current implementation does not scale to provide guarantees for real-world neural networks.

Machine Learning,Logic in Computer Science
Towards Testing of Deep Learning Systems with Training Set Reduction

Helge Spieker,Arnaud Gotlieb

DOI: https://doi.org/10.48550/arXiv.1901.04169

2019-01-14

Abstract:Testing the implementation of deep learning systems and their training routines is crucial to maintain a reliable code base. Modern software development employs processes, such as Continuous Integration, in which changes to the software are frequently integrated and tested. However, testing the training routines requires running them and fully training a deep learning model can be resource-intensive, when using the full data set. Using only a subset of the training data can improve test run time, but can also reduce its effectiveness. We evaluate different ways for training set reduction and their ability to mimic the characteristics of model training with the original full data set. Our results underline the usefulness of training set reduction, especially in resource-constrained environments.

Machine Learning,Neural and Evolutionary Computing,Software Engineering
Simplification of Polyhedral Reductions in Practice

Louis Narmour,Ryan Job,Tomofumi Yuki,Sanjay Rajopadhye

2024-11-26

Abstract:Reductions combine collections of inputs with an associative (and here, also commutative) operator to produce collections of outputs. When the same value contributes to multiple outputs, there is an opportunity to reuse partial results, enabling reduction simplification. We provide the first complete push-button implementation of reduction simplification in a compiler. We evaluate its effectiveness on a range of real-world applications, and show that simplification rediscovers several key results in algorithmic improvement across multiple domains, previously only obtained through clever manual human analysis and effort. We also discover alternate, previously unknown algorithms, albeit without improving the asymptotic complexity.

Programming Languages

Learning Reductions that Really Work

Tool-Assisted Learning of Computational Reductions

Verified reductions for optimization

Optimality-based domain reduction for inequality-constrained NLP and MINLP problems

Knowledge Reduction in Real Decision Formal Contexts

The Neural Race Reduction: Dynamics of Abstraction in Gated Networks

Logical reduction of metarules

Experimental Comparisons of Instances Set Reduction Algorithms

Fully Automatic Neural Network Reduction for Formal Verification

Impact of domain reduction techniques in polynomial optimization: A computational study

Learning with Known Operators reduces Maximum Training Error Bounds

Knowledge Reduction for Formal Context Based on Concept Lattice

On the convergence of reduction-based and model-based methods in proof theory

LeanAgent: Lifelong Learning for Formal Theorem Proving

Compressing neural networks via formal methods

Achieving Occam's Razor: Deep Learning for Optimal Model Reduction

SiMpLIfy: A Toolbox for Structured Model Reduction

Learning advanced mathematical computations from examples

Machine Learning with Guarantees using Descriptive Complexity and SMT Solvers

Towards Testing of Deep Learning Systems with Training Set Reduction

Simplification of Polyhedral Reductions in Practice