Abstract:This paper aims to explain how a deep neural network (DNN) gradually extracts new knowledge and forgets noisy features through layers in forward propagation. Up to now, although the definition of knowledge encoded by the DNN has not reached a consensus, Previous studies have derived a series of mathematical evidence to take interactions as symbolic primitive inference patterns encoded by a DNN. We extend the definition of interactions and, for the first time, extract interactions encoded by intermediate layers. We quantify and track the newly emerged interactions and the forgotten interactions in each layer during the forward propagation, which shed new light on the learning behavior of DNNs. The layer-wise change of interactions also reveals the change of the generalization capacity and instability of feature representations of a DNN.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to explain how deep neural networks (DNNs) progressively extract new knowledge and forget noisy features during the forward propagation process. Specifically: 1. **Definition and Quantification of Knowledge**: - The authors extend the definition of "interactions" and, for the first time, extract interactions from intermediate layers. - Researchers quantify and track the newly emerging and forgotten interactions in each layer, revealing the learning behavior of DNNs. 2. **Challenges of Knowledge Change**: - Three main challenges are proposed: alignment of interaction primitives, decomposability and countability of knowledge, and its connection to generalization ability. - By training linear classifiers to extract interactions from each layer and analyzing the fidelity of these interactions, these challenges are addressed. 3. **Fidelity of Interactions**: - The newly defined interaction primitives still belong to the typical interaction paradigm, and their fidelity can be proven through a series of theorems. 4. **Contributions**: - Redefines interactions on intermediate layers and finds that adjacent layers encode similar interactions. - Provides several theoretically verifiable metrics to quantify the newly emerging and forgotten knowledge during the forward propagation process. - Discovers that changes in interactions are also related to the generalization ability of DNNs. In summary, this paper is dedicated to revealing the learning behavior of DNNs by quantifying and tracking interactions in various layers and further understanding the generalization ability of DNNs.

Layerwise Change of Knowledge in Neural Networks

Towards the Dynamics of a DNN Learning Symbolic Interactions

Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification

Deep Neural Networks With Knowledge Instillation

Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features

Biologically Inspired Structure Learning with Reverse Knowledge Distillation for Spiking Neural Networks

Interpretability of Neural Networks Based on Game-theoretic Interactions

A Deeper Knowledge Tracking Model Integrating Cognitive Theory and Learning Behavior

Explaining Knowledge Distillation by Quantifying the Knowledge

Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning

The two-way knowledge interaction interface between humans and neural networks

Towards the Difficulty for a Deep Neural Network to Learn Concepts of Different Complexities

Going Deeper, Generalizing Better: an Information-Theoretic View for Deep Learning.

Explaining Generalization Power of a DNN Using Interactive Concepts

Towards Neural Knowledge DNA

Worth of knowledge in deep learning

EGNN: Constructing explainable graph neural networks via knowledge distillation

Does a Neural Network Really Encode Symbolic Concepts?

What does the Knowledge Neuron Thesis Have to do with Knowledge?

Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting

Defining and Extracting generalizable interaction primitives from DNNs