Abstract:This paper proposes a new approach to Machine Learning (ML) that focuses on unsupervised continuous context-dependent learning of complex patterns. Although the proposal is partly inspired by some of the current knowledge about the structural and functional properties of the mammalian brain, we do not claim that biological systems work in an analogous way (nor the opposite). Based on some properties of the cerebellar cortex and adjacent structures, a proposal suitable for practical problems is presented. A synthetic structure capable of identifying and predicting complex temporal series will be defined and experimentally tested. The system relies heavily on prediction to help identify and learn patterns based on previously acquired contextual knowledge. As a proof of concept, the proposed system is shown to be able to learn, identify and predict a remarkably complex temporal series such as human speech, with no prior knowledge. From raw data, without any adaptation in the core algorithm, the system is able to identify certain speech structures from a set of Spanish sentences. Unlike conventional ML, the proposal can learn with a reduced training set. Although the idea can be applied to a constrained problem, such as the detection of unknown vocabulary in a speech, it could be used in more applications, such as vision, or (by incorporating the missing biological periphery) fit into other ML techniques. Given the trivial computational primitives used, a potential hardware implementation will be remarkably frugal. Coincidentally, the proposed model not only conforms to a plausible functional framework for biological systems but may also explain many elusive cognitive phenomena.
Neural and Evolutionary Computing,Artificial Intelligence,Hardware Architecture,Machine Learning
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is to develop a new machine - learning method that can perform unsupervised continuous context - dependent learning, especially for the recognition and prediction of complex patterns. The core objective of the paper is to design a synthetic structure that can recognize and predict complex time - series data, such as human languages, without any prior knowledge. The key to this system lies in using prediction to help recognize and learn patterns, based on the previously obtained context knowledge.
Specifically, the paper proposes an architecture named Hierarchical Event Recaller (HER), whose design is partly inspired by the structural and functional characteristics of the mammalian brain, especially certain properties of the cerebellar cortex and its adjacent structures. HER achieves its goals through the following core principles:
1. **Time Prediction Based on Sequence Segmentation**: The core component of HER is a biological sequence memory, which can recognize and mark paragraphs or sequences in the input stream at each level of the hierarchical structure. Each recognition result will be passed to the next level (called "rung") in the hierarchical structure, where sequences with higher semantic significance will be learned, recognized, and predicted. By segmenting the input stream at each level, HER can process very complex patterns with limited resources (i.e., cells and synapses).
2. **Short - term and Long - term Learning Regulation**: To ensure the flexibility of the system, the learning process is stochastic. The probability of learning is determined by the knowledge level of the current input sequence. If the recent sequence value is predicted wrongly (i.e., the sequence is unknown in the short term), the probability of changing synaptic persistence will increase significantly; otherwise, this probability will be very low. The sequence memory groups within each cortical column (CC) will be in one of two states, known or unknown, based on the input activity several time steps ago. Similarly, in a longer time range, the entire CC will also have a corresponding state. Learning can be enhanced or weakened as needed, depending on the CC's long - term familiarity with the input.
3. **Input Dimensionality Reduction, Pattern Disambiguation, and Feedback**: Each sequence memory processes a limited number of input values. By using the k - winner - take - all inhibition mechanism, the input dimension is reduced. Many silent synapses in the network project each input value into a few cluster representations, which are associated with symbols. The sequence memory can accurately track the time context of each symbol (using a state representation based on combinatorics). Duplication can be used to improve the robustness of the system or promote the disambiguation of complex patterns. Internal feedback helps to stabilize the inhibition process (i.e., stabilize symbol generation).
4. **False Recognition Filtering, Learning Acceleration, and Learning Gating**: During the learning process, an auxiliary sequence memory module (similar to part of the function of the hippocampus) is responsible for preventing higher - level hierarchies from being contaminated by false or incomplete recognitions. This module is required at all levels and can be statically allocated or allocated on - demand from a centralized structure. When the long - term knowledge state of a CC in the next level is unknown, this module will be allocated. If this module is not familiar with the current data, the learning of the CC in the next level will be disabled. Once all CCs in the next level are familiar with the data, this component will be released.
5. **Lateral Contextualization**: Each CC will have two types of inputs: (1) vertical input from a single predecessor cortical column (CCprev), which is used to form the time reference frame of the sequence; (2) lateral input from neighboring columns, which is used to uniquely identify the sequence in the spatial context of CCprev. Combining vertical and horizontal inputs, the CC will generate a single recognition result within the reference frame constructed by vertical connections, which is modulated by the sequences perceived by neighboring CCs.
6. **Attention and Speculative Pattern Recognition**: Each CC has the ability to speculatively predict the input sequence recognition in higher levels. If the expectation at the higher level (i.e., the provided feedback) matches some form of advanced prediction in the current level, the sequence can be predicted in advance without fully receiving it. This will accelerate the recognition process and help compress similar sequences into a common recognition result.
7. **Supervised Learning (and Output Stabilization)**: The system can incorporate mechanisms to guide the learning output of each CC to reach specific goals, either temporally or representationally. This can also be used to stabilize the output of the CC in the long - term known state. Although this learning/recognition goal can be self - generated, the proposed system lacks a mechanism for generating these goals. It may be necessary to introduce action capabilities (e.g., changing sensors) and/or certain reward - based selection strategies.
Through this...