A geometrical representation of McCulloch-Pitts neural model and its applications

ZhangLing,ZhangBo

1999-01-01

IEEE Transactions on Neural Networks

Abstract:In this paper, a geometrical representation of McCulloch-Pitts neural model (1943) is presented, From the representation, a clear visual picture and interpretation of the model can be seen. Two int...

What problem does this paper attempt to address?

A Geometrical Representation of M-P Neural Model and Its Applications

ZHANG Ling,ZHANG Bo

1998-01-01

Journal of Software

Abstract:In this paper, a geometrical representation of M P neural model is presented. From the representation,a clear visual picture and interpretation of the model can be seen. Two interesting applications based on the interpretation are discussed. They are (1) a new design principle of feedforward neural networks, and (2) a new proof of mapping abilities of three layer feedforward neural networks.
Neural Network Based Classifers for a Vast Amount of Data

Ling Zhang,Bo Zhang

DOI: https://doi.org/10.1007/3-540-48912-6_32

1999-01-01

Abstract:When using neural networks to train a large number of data for classification, there generally exists a learning complexity problem. In this paper, a new geometrical interpretation of McCulloch-Pitts (M-P) neural model is presented. Based on the interpretation, a new constructive learning approach is discussed. Experimental results show that the new algorithm can greatly reduce the learning complexity and can be applied to real classification problems with a vast amount of data.
Visualizing and Understanding Neural Models in NLP

Jiwei Li,Xinlei Chen,Eduard Hovy,Dan Jurafsky

DOI: https://doi.org/10.18653/v1/n16-1082

2015-01-01

Abstract:While neural networks have been successfully applied to many NLP tasks the resulting vector-based models are very difficult to interpret. For example it's not clear how they achieve {\em compositionality}, building sentence meaning from the meanings of words and phrases. In this paper we describe four strategies for visualizing compositionality in neural models for NLP, inspired by similar work in computer vision. We first plot unit values to visualize compositionality of negation, intensification, and concessive clauses, allow us to see well-known markedness asymmetries in negation. We then introduce three simple and straightforward methods for visualizing a unit's {\em salience}, the amount it contributes to the final composed meaning: (1) gradient back-propagation, (2) the variance of a token from the average word node, (3) LSTM-style gates that measure information flow. We test our methods on sentiment using simple recurrent nets and LSTMs. Our general-purpose methods may have wide applications for understanding compositionality and other semantic properties of deep networks , and also shed light on why LSTMs outperform simple recurrent nets,
McCulloch-Pitts brains and pseudorandom functions

Vašek Chvátal,Mark Goldsmith,Nan Yang

DOI: https://doi.org/10.48550/arXiv.1603.01573

2016-03-05

Abstract:In a pioneering classic, Warren McCulloch and Walter Pitts proposed a model of the central nervous system. Motivated by EEG recordings of normal brain activity, Chvátal and Goldsmith asked whether or not these dynamical systems can be engineered to produce trajectories which are irregular, disorderly, apparently unpredictable. We show that they cannot build weak pseudorandom functions.

Dynamical Systems,Cryptography and Security
An Exhaustive Study of Two-Node McCulloch-Pitts Networks

Wentian Li,Astero Provata,Thomas MacCarthy

2024-06-29

Abstract:Boolean networks are widely used in computational biology, evolutionary studies, and social sciences. However, the set of all Boolean-function-defined networks are harder to study as a whole. On the other hand, McCulloch-Pitts gates are sparsely parameterized using only a few number of link strengths, making it possible to study and compare different networks models. We treat two-node McCulloch-Pitts systems as a minimal complex system. When the link strengths are discretized, $3^4=81$ network models or rules are organized in the rule space The limiting dynamics of each rule may depend on the choice of binary state value ([-1,1] or [0,1]), and on the treatment at the threshold point, leading to at least six variants. One variant with [-1,1] as the binary state value (V1 model) tends to have a more diverse dynamical behaviors with a mixture of multiple cycles and fixed points at the limiting state, whereas other variants tend to fall only to fixed-point limiting dynamics. We use V1 models to study the organization of rules with different dynamics in the rule space and robustness of limiting dynamics with respect to a mutation in the rule table, as well as the related phenomena of phase transition and edge-of-chaos. We use another variant (V4 models) with only the fixed-point limiting dynamics to study the robustness of limiting state with respect to perturbation of initial states. The two types of robustness do not seem to be associated with each other. Other aspects of fully discretized two-node MaCulloch-Pitts networks are also studied, including: the proposal of a seventh variant based on a difference equation; relation to Rene Thomas' two types of feedback loops; spectrum properties of state space transition matrix; and asynchronous updating. Our works also expand the concept of network motifs by allowing more finer details.

Social and Information Networks,Physics and Society
Neural population geometry: An approach for understanding biological and artificial neural networks

SueYeon Chung,L. F. Abbott

DOI: https://doi.org/10.1016/j.conb.2021.10.010

2021-11-20

Abstract:Advances in experimental neuroscience have transformed our ability to explore the structure and function of neural circuits. At the same time, advances in machine learning have unleashed the remarkable computational power of artificial neural networks (ANNs). While these two fields have different tools and applications, they present a similar challenge: namely, understanding how information is embedded and processed through high-dimensional representations to solve complex tasks. One approach to addressing this challenge is to utilize mathematical and computational tools to analyze the geometry of these high-dimensional representations, i.e., neural population geometry. We review examples of geometrical approaches providing insight into the function of biological and artificial neural networks: representation untangling in perception, a geometric theory of classification capacity, disentanglement and abstraction in cognitive systems, topological representations underlying cognitive maps, dynamic untangling in motor systems, and a dynamical approach to cognition. Together, these findings illustrate an exciting trend at the intersection of machine learning, neuroscience, and geometry, in which neural population geometry provides a useful population-level mechanistic descriptor underlying task implementation. Importantly, geometric descriptions are applicable across sensory modalities, brain regions, network architectures and timescales. Thus, neural population geometry has the potential to unify our understanding of structure and function in biological and artificial neural networks, bridging the gap between single neurons, populations and behavior.

Neurons and Cognition,Machine Learning
A Bio-inspired Model for Image Representation and Image Analysis

Hui Wei,Qingsong Zuo,Bo Lang

DOI: https://doi.org/10.1109/ICTAI.2011.67

2011-01-01

Abstract:This paper proposes a model for image representation and image analysis using a multi-layer neural network, which is rooted in the human vision system. Having complex neural layers to represent and process information, the biological vision system is far more efficient than machine vision system. The neural model simulate non-classical receptive field of ganglion cell and its local feedback control circuit, and can represent images, beyond pixel level, self-adaptively and regularly. The results of experiments, rebuilding, distribution and contour detection, prove this method can represent image faithfully with low cost, and can produce a compact and abstract approximation to facilitate successive image segmentation and integration. This representation schema is good at extracting spatial relationships from different components of images and highlighting foreground objects from background, especially for nature images with complicated scenes. Further it can be applied to object recognition or image classification tasks in future.
Geometrization of deep networks for the interpretability of deep learning systems

Xiao Dong,Ling Zhou

DOI: https://doi.org/10.48550/arXiv.1901.02354

IF: 5.414

2019-01-06

Machine Learning

Abstract:How to understand deep learning systems remains an open problem. In this paper we propose that the answer may lie in the geometrization of deep networks. Geometrization is a bridge to connect physics, geometry, deep network and quantum computation and this may result in a new scheme to reveal the rule of the physical world. By comparing the geometry of image matching and deep networks, we show that geometrization of deep networks can be used to understand existing deep learning systems and it may also help to solve the interpretability problem of deep learning systems.
A structural representing and learning model based on biological neural mechanism

Hui Wei,HuiXuan Tang

DOI: https://doi.org/10.1109/ICMLC.2004.1384588

2004-01-01

Abstract:From the view of cognitive computational neuroscience, a direct representing method based on neural dynamics and graph theory is presented. Firstly, an assembly of neuron as well as its dynamics is defined. They directly represent the perceptual information of stimulus. Then a two layer neural network is designed to retain the features of that stimulus and generate the very neural circuit responding to it. This is achieved by the structure-learning algorithm. The circuit can also serve as an associative base whose credibility is decided by its connectivity. The direct representing method is of great significance in the research of semantic representation and semantic-driven inference in artificial intelligence as well as in artificial neural network researches. The exhibition of major physiological features of neural information processing distinguishes this model from the traditional ones.
Revisiting Marr in Face: The Building of 2D--2.5D--3D Representations in Deep Neural Networks

Xiangyu Zhu,Chang Yu,Jiankuo Zhao,Zhaoxiang Zhang,Stan Z. Li,Zhen Lei

2024-11-25

Abstract:David Marr's seminal theory of vision proposes that the human visual system operates through a sequence of three stages, known as the 2D sketch, the 2.5D sketch, and the 3D model. In recent years, Deep Neural Networks (DNN) have been widely thought to have reached a level comparable to human vision. However, the mechanisms by which DNNs accomplish this and whether they adhere to Marr's 2D--2.5D--3D construction theory remain unexplored. In this paper, we delve into the perception task to explore these questions and find evidence supporting Marr's theory. We introduce a graphics probe, a sub-network crafted to reconstruct the original image from the network's intermediate layers. The key to the graphics probe is its flexible architecture that supports image in both 2D and 3D formats, as well as in a transitional state between them. By injecting graphics probes into neural networks, and analyzing their behavior in reconstructing images, we find that DNNs initially encode images as 2D representations in low-level layers, and finally construct 3D representations in high-level layers. Intriguingly, in mid-level layers, DNNs exhibit a hybrid state, building a geometric representation that s sur normals within a narrow depth range, akin to the appearance of a low-relief sculpture. This stage resembles the 2.5D representations, providing a view of how DNNs evolve from 2D to 3D in the perception process. The graphics probe therefore serves as a tool for peering into the mechanisms of DNN, providing empirical support for Marr's theory.

Computer Vision and Pattern Recognition
A General Image Representation Scheme and Its Improvement for Image Analysis

Hui Wei,Qingsong Zuo,Bo Lang

DOI: https://doi.org/10.1007/978-3-642-40728-4_45

2014-01-01

Abstract:In this paper, a bio-inspired neural network is developed to represent images and analysis features of images effectively. This model adopts schemes of retinal ganglion cells (GC) working and GCs’ non-classical receptive fields (nCRF) that can dynamically adjust their sizes/scales according to the visual information. Extensive experiments are provided to value the effect of image representing, and experimental results show that this neural network model can represent images at a low cost and with a favor in improving both segmentation and integration processing. Most importantly, the GC-array model provides a basic infrastructure for image semantic extraction.
A mathematical model of the visual MacKay effect

Cyprien Tamekue,Dario Prandi,Yacine Chitour

2024-08-02

Abstract:This paper investigates the intricate connection between visual perception and the mathematical modeling of neural activity in the primary visual cortex (V1). The focus is on modeling the visual MacKay effect [D. M. MacKay, Nature, 180 (1957), pp. 849--850]. While bifurcation theory has been a prominent mathematical approach for addressing issues in neuroscience, especially in describing spontaneous pattern formations in V1 due to parameter changes, it faces challenges in scenarios with localized sensory inputs. This is evident, for instance, in MacKay's psychophysical experiments, where the redundancy of visual stimuli information results in irregular shapes, making bifurcation theory and multiscale analysis less effective. To address this, we follow a mathematical viewpoint based on the input-output controllability of an Amari-type neural fields model. In this framework, we consider sensory input as a control function, a cortical representation via the retino-cortical map of the visual stimulus that captures its distinct features. This includes highly localized information in the center of MacKay's funnel pattern "MacKay rays". From a control theory point of view, the Amari-type equation's exact controllability property is discussed for linear and nonlinear response functions. For the visual MacKay effect modeling, we adjust the parameter representing intra-neuron connectivity to ensure that cortical activity exponentially stabilizes to the stationary state in the absence of sensory input. Then, we perform quantitative and qualitative studies to demonstrate that they capture all the essential features of the induced after-image reported by MacKay.

Optimization and Control,Analysis of PDEs,Numerical Analysis,Neurons and Cognition
The Representation Theory of Neural Networks

Marco Armenta,Pierre-Marc Jodoin

DOI: https://doi.org/10.3390/math9243216

IF: 2.4

2021-12-13

Mathematics

Abstract:In this work, we show that neural networks can be represented via the mathematical theory of quiver representations. More specifically, we prove that a neural network is a quiver representation with activation functions, a mathematical object that we represent using a network quiver. Furthermore, we show that network quivers gently adapt to common neural network concepts such as fully connected layers, convolution operations, residual connections, batch normalization, pooling operations and even randomly wired neural networks. We show that this mathematical representation is by no means an approximation of what neural networks are as it exactly matches reality. This interpretation is algebraic and can be studied with algebraic methods. We also provide a quiver representation model to understand how a neural network creates representations from the data. We show that a neural network saves the data as quiver representations, and maps it to a geometrical space called the moduli space, which is given in terms of the underlying oriented graph of the network, i.e., its quiver. This results as a consequence of our defined objects and of understanding how the neural network computes a prediction in a combinatorial and algebraic way. Overall, representing neural networks through the quiver representation theory leads to 9 consequences and 4 inquiries for future research that we believe are of great interest to better understand what neural networks are and how they work.

mathematics
A neural geometry theory comprehensively explains apparently conflicting models of visual perceptual learning

Yu-Ang Cheng Sr.,Mehdi Sanayei Sr.,Xing Chen,Ke Jia,Sheng Li,Fang Fang,Takeo Watanabe,Alexander Thiele,Ru-Yuan Zhang

DOI: https://doi.org/10.1101/2023.11.13.566963

2024-05-19

Abstract:Visual perceptual learning (VPL), defined as long-term improvement in a visual task, is considered a crucial tool for elucidating underlying visual and brain plasticity. However, the identification of a unified theory of VPL has long been controversial. Multiple existing models have proposed diverse mechanisms, including improved signal-to-noise ratio, changes in tuning curves, and reduction of noise correlations, as major contributors to improved neural representations associated with VPL. However, each model only accounts for specific aspects of the empirical findings, and there exists no theory that can comprehensively explain all empirical results. Here, we argue that all neural changes at single units can be conceptualized as geometric transformations of population response manifolds in a high-dimensional neural space. This approach enables conflicting major models of VPL to be quantitatively tested and compared within a unified computational theory. Following this approach, we found that changes in tuning curves and noise correlations, as emphasized by previous models, make no significant contributions to improved population representations by visual training. Instead, we identified neural manifold shrinkage due to reduced trial-by-trial neural response variability, a previously unexplored factor, as the primary mechanism underlying improved population representations. Furthermore, we showed that manifold shrinkage successfully accounts for learning effects across various domains, including artificial neural responses in deep neural networks trained on typical VPL tasks, multivariate BOLD signals in humans, and multi-unit activities in monkeys. These converging results suggest that our neural geometry theory offers a quantitative and comprehensive approach to explain a wide range of empirical results and to reconcile previously conflicting models of VPL.

Neuroscience
Understanding over-parameterized deep networks by geometrization

Xiao Dong,Ling Zhou

DOI: https://doi.org/10.48550/arXiv.1902.03793

2019-02-11

Abstract:A complete understanding of the widely used over-parameterized deep networks is a key step for AI. In this work we try to give a geometric picture of over-parameterized deep networks using our geometrization scheme. We show that the Riemannian geometry of network complexity plays a key role in understanding the basic properties of over-parameterizaed deep networks, including the generalization, convergence and parameter sensitivity. We also point out deep networks share lots of similarities with quantum computation systems. This can be regarded as a strong support of our proposal that geometrization is not only the bible for physics, it is also the key idea to understand deep learning systems.

Machine Learning
PottsMGNet: A Mathematical Explanation of Encoder-Decoder Based Neural Networks

Xue-Cheng Tai,Hao Liu,Raymond Chan

DOI: https://doi.org/10.1137/23m1586355

IF: 1.938

2024-03-09

SIAM Journal on Imaging Sciences

Abstract:SIAM Journal on Imaging Sciences, Volume 17, Issue 1, Page 540-594, March 2024. For problems in image processing and many other fields, a large class of effective neural networks has encoder-decoder-based architectures. Although these networks have shown impressive performance, mathematical explanations of their architectures are still underdeveloped. In this paper, we study the encoder-decoder-based network architecture from the algorithmic perspective and provide a mathematical explanation. We use the two-phase Potts model for image segmentation as an example for our explanations. We associate the segmentation problem with a control problem in the continuous setting. Then, the continuous control model is time discretized by an operator-splitting scheme, the PottsMGNet, and space discretized by the multigrid method. We show that the resulting discrete PottsMGNet is equivalent to an encoder-decoder-based network. With minor modifications, it is shown that a number of the popular encoder-decoder-based neural networks are just instances of the proposed PottsMGNet. By incorporating the soft-threshold-dynamics into the PottsMGNet as a regularizer, the PottsMGNet has shown to be robust with the network parameters such as network width and depth and has achieved remarkable performance on datasets with very large noise. In nearly all our experiments, the new network always performs better than or as well as on accuracy and dice score compared to existing networks for image segmentation.

mathematics, applied,computer science, software engineering, artificial intelligence,imaging science & photographic technology
A Fuzzy Classifier Based on the Constructive Covering Approach in Neural Networks

叶少珍,张钹,吴鸣锐,郑文波

2003-01-01

Ruan Jian Xue Bao/Journal of Software

Abstract:A geometrical representation of M-P model is firstly introduced, by which the training problem of neural networks may be transformed into the covering problem of a point set. According to this, the geometrical algorithm of neural network training is analyzed. The algorithm may be used for constructing very complicated classifying boundary, but it has higher time complexity. So a fuzzy classifier based on the combination of the covering approach and fuzzy set theory is proposed. The classifier can improve the speed of training and decrease the number of covering sphere-neighborhoods, i.e., decrease the number of hidden nodes of neural networks. The fuzzy set based approach may also provide multi-choices for pattern recognition problems of large scale. Recognition of 700 handwritten Chinese characters is used to test the performance of the approach and the results are promising.
An Interpretable Neural Network Model Through Piecewise Linear Approximation

Mengzhuo Guo,Qingpeng Zhang,Xiuwu Liao,Daniel Dajun Zeng

DOI: https://doi.org/10.48550/arxiv.2001.07119

2020-01-01

Abstract:Most existing interpretable methods explain a black-box model in a post-hoc manner, which uses simpler models or data analysis techniques to interpret the predictions after the model is learned. However, they (a) may derive contradictory explanations on the same predictions given different methods and data samples, and (b) focus on using simpler models to provide higher descriptive accuracy at the sacrifice of prediction accuracy. To address these issues, we propose a hybrid interpretable model that combines a piecewise linear component and a nonlinear component. The first component describes the explicit feature contributions by piecewise linear approximation to increase the expressiveness of the model. The other component uses a multi-layer perceptron to capture feature interactions and implicit nonlinearity, and increase the prediction performance. Different from the post-hoc approaches, the interpretability is obtained once the model is learned in the form of feature shapes. We also provide a variant to explore higher-order interactions among features to demonstrate that the proposed model is flexible for adaptation. Experiments demonstrate that the proposed model can achieve good interpretability by describing feature shapes while maintaining state-of-the-art accuracy.
Klein Model for Hyperbolic Neural Networks

Yidan Mao,Jing Gu,Marcus C. Werner,Dongmian Zou

2024-10-22

Abstract:Hyperbolic neural networks (HNNs) have been proved effective in modeling complex data structures. However, previous works mainly focused on the Poincaré ball model and the hyperboloid model as coordinate representations of the hyperbolic space, often neglecting the Klein model. Despite this, the Klein model offers its distinct advantages thanks to its straight-line geodesics, which facilitates the well-known Einstein midpoint construction, previously leveraged to accompany HNNs in other models. In this work, we introduce a framework for hyperbolic neural networks based on the Klein model. We provide detailed formulation for representing useful operations using the Klein model. We further study the Klein linear layer and prove that the "tangent space construction" of the scalar multiplication and parallel transport are exactly the Einstein scalar multiplication and the Einstein addition, analogous to the Möbius operations used in the Poincaré ball model. We show numerically that the Klein HNN performs on par with the Poincaré ball model, providing a third option for HNN that works as a building block for more complicated architectures.

Machine Learning,Information Theory
Visualizing a Neural Network That Develops Quantum Perturbation Theory

Yadong Wu,Pengfei Zhang,Huitao Shen,Hui Zhai

DOI: https://doi.org/10.1103/physreva.98.010701

IF: 8.6

2018-01-01

Physical Review Letters

Abstract:Motivated by the question whether the empirical fitting of data by neural networks can yield the same structure of physical laws, we apply neural networks to a quantum-mechanical two-body scattering problem with short-range potentials-a problem that by itself plays an important role in many branches of physics. After training, the neural network can accurately predict s-wave scattering length, which governs the low-energy scattering physics. By visualizing the neural network, we show that it develops perturbation theory order by order when the potential depth increases, without solving the Schrodinger equation or obtaining the wave function explicitly. The result provides an important benchmark to the machine-assisted physics research or even automated machine learning physics laws.

A geometrical representation of McCulloch-Pitts neural model and its applications

A Geometrical Representation of M-P Neural Model and Its Applications

Neural Network Based Classifers for a Vast Amount of Data

Visualizing and Understanding Neural Models in NLP

McCulloch-Pitts brains and pseudorandom functions

An Exhaustive Study of Two-Node McCulloch-Pitts Networks

Neural population geometry: An approach for understanding biological and artificial neural networks

A Bio-inspired Model for Image Representation and Image Analysis

Geometrization of deep networks for the interpretability of deep learning systems

A structural representing and learning model based on biological neural mechanism

Revisiting Marr in Face: The Building of 2D--2.5D--3D Representations in Deep Neural Networks

A General Image Representation Scheme and Its Improvement for Image Analysis

A mathematical model of the visual MacKay effect

The Representation Theory of Neural Networks

A neural geometry theory comprehensively explains apparently conflicting models of visual perceptual learning

Understanding over-parameterized deep networks by geometrization

PottsMGNet: A Mathematical Explanation of Encoder-Decoder Based Neural Networks

A Fuzzy Classifier Based on the Constructive Covering Approach in Neural Networks

An Interpretable Neural Network Model Through Piecewise Linear Approximation

Klein Model for Hyperbolic Neural Networks

Visualizing a Neural Network That Develops Quantum Perturbation Theory