On the Role of Noise in Factorizers for Disentangling Distributed Representations

Geethan Karunaratne,Michael Hersche,Abu Sebastian,Abbas Rahimi

2024-11-30

Abstract:To efficiently factorize high-dimensional distributed representations to the constituent atomic vectors, one can exploit the compute-in-superposition capabilities of vector-symbolic architectures (VSA). Such factorizers however suffer from the phenomenon of limit cycles. Applying noise during the iterative decoding is one mechanism to address this issue. In this paper, we explore ways to further relax the noise requirement by applying noise only at the time of VSA's reconstruction codebook initialization. While the need for noise during iterations proves analog in-memory computing systems to be a natural choice as an implementation media, the adequacy of initialization noise allows digital hardware to remain equally indispensable. This broadens the implementation possibilities of factorizers. Our study finds that while the best performance shifts from initialization noise to iterative noise as the number of factors increases from 2 to 4, both extend the operational capacity by at least 50 times compared to the baseline factorizer resonator networks. Our code is available at: <a class="link-external link-https" href="https://github.com/IBM/in-memory-factorizer" rel="external noopener nofollow">this https URL</a>

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper explores how to solve the limit cycles problem in factorizers by introducing noise, thereby enhancing the disentanglement ability of distributed representations. Specifically:

1. **Limit Cycles Problem**:
- When factorizers are dealing with high - dimensional distributed representations, they are prone to fall into limit cycles. That is, during the iteration process, the state of the network will oscillate repeatedly within a small range and cannot converge to the correct solution.
- This phenomenon is particularly evident in resonator networks based on vector symbolic architectures (VSA), because these networks use symmetric and deterministic codebooks in the association search (AS) and reconstruction (RC) stages, resulting in deterministic behavior.

2. **The Role of Introducing Noise**:
- The paper proposes to break this determinism by introducing noise, enabling the network to jump out of the limit cycles and explore a wider solution space.
- Specific methods include:
- Introducing iterative noise during each iteration process. For example, the IMF (In - Memory Factorizer) method utilizes the inherent noise in in - memory computing.
- Only introducing initialization noise when initializing the codebook. For example, the ACF (Asymmetric Codebook Factorizer) method perturbs the codebook through a bit - flip mask.

3. **Options for Hardware Implementation**:
- Introducing noise not only improves the operational capacity of factorizers but also affects their hardware implementation methods. For example, the IMF method is suitable for in - memory computing systems, while the ACF method is applicable to digital hardware designs.

4. **Performance Improvement**:
- The experimental results show that, compared with the baseline resonator network (BRN), both IMF and ACF have significant improvements in operational capacity and convergence speed. Especially in cases with different numbers of factors (F = 2, 3, 4), both methods can handle larger - scale search spaces and require fewer iterations.

In summary, this paper aims to overcome the limit cycles problem in factorizers by introducing noise, thereby enhancing their disentanglement ability and operational capacity, and providing new ideas for different hardware implementations.