Abstract:Neural networks trained to solve modular arithmetic tasks exhibit grokking, a phenomenon where the test accuracy starts improving long after the model achieves 100% training accuracy in the training process. It is often taken as an example of "emergence", where model ability manifests sharply through a phase transition. In this work, we show that the phenomenon of grokking is not specific to neural networks nor to gradient descent-based optimization. Specifically, we show that this phenomenon occurs when learning modular arithmetic with Recursive Feature Machines (RFM), an iterative algorithm that uses the Average Gradient Outer Product (AGOP) to enable task-specific feature learning with general machine learning models. When used in conjunction with kernel machines, iterating RFM results in a fast transition from random, near zero, test accuracy to perfect test accuracy. This transition cannot be predicted from the training loss, which is identically zero, nor from the test loss, which remains constant in initial iterations. Instead, as we show, the transition is completely determined by feature learning: RFM gradually learns block-circulant features to solve modular arithmetic. Paralleling the results for RFM, we show that neural networks that solve modular arithmetic also learn block-circulant features. Furthermore, we present theoretical evidence that RFM uses such block-circulant features to implement the Fourier Multiplication Algorithm, which prior work posited as the generalizing solution neural networks learn on these tasks. Our results demonstrate that emergence can result purely from learning task-relevant features and is not specific to neural architectures nor gradient descent-based optimization methods. Furthermore, our work provides more evidence for AGOP as a key mechanism for feature learning in neural networks.

Machine learning for modular multiplication

Revisiting Scalable Modular Multiplication over GF(2/sup M/) for Elliptic Curve Cryptography

Unbalanced Exponent Modular Reduction over Binary Field and Its Implementation

Research of Fast Modular Multiplier for a Class of Finite Fields

Teaching Transformers Modular Arithmetic at Scale

Probability Method for Cryptanalysis of General Multivariate Modular Linear Equation

Fast Modular Multiplication And Parallel Algorithms In Public-Key Cryptosystems

Machine learning for moduli space of genus two curves and an application to isogeny based cryptography

Grokking Modular Polynomials

Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs

Simultaneous Modular Reduction and Kronecker Substitution for Small Finite Fields

Low-Latency Scalable Modular Multiplier Without Montgomery Algorithm

Secure Computation for G-Module and Its Applications

Long Polynomial Modular Multiplication using Low-Complexity Number Theoretic Transform

Design of a Modular Multiplier for ECC Cryptosystem

Emergence in non-neural models: grokking modular arithmetic via average gradient outer product

A Novel Method of Modular Multiplication Based on Karatsuba-like Multiplication

A DPA-Resistant Digit-Parallel Modular Multiplier over GF (2m)

Applications of Machine Learning in Cryptography: A Survey

Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks

Applications to Montgomery Modular Multiplication of Karatsuba Multiplication Without Overlapped Summation