Weight Freezing: A Regularization Approach for Fully Connected Layers with an Application in EEG Classification

Zhengqing Miao,Meirong Zhao
2023-06-12
Abstract:In the realm of EEG decoding, enhancing the performance of artificial neural networks (ANNs) carries significant potential. This study introduces a novel approach, termed "weight freezing", that is anchored on the principles of ANN regularization and neuroscience prior knowledge. The concept of weight freezing revolves around the idea of reducing certain neurons' influence on the decision-making process for a specific EEG task by freezing specific weights in the fully connected layer during the backpropagation process. This is actualized through the use of a mask matrix and a threshold to determine the proportion of weights to be frozen during backpropagation. Moreover, by setting the masked weights to zero, weight freezing can not only realize sparse connections in networks with a fully connected layer as the classifier but also function as an efficacious regularization method for fully connected layers. Through experiments involving three distinct ANN architectures and three widely recognized EEG datasets, we validate the potency of weight freezing. Our method significantly surpasses previous peak performances in classification accuracy across all examined datasets. Supplementary control experiments offer insights into performance differences pre and post weight freezing implementation and scrutinize the influence of the threshold in the weight freezing process. Our study underscores the superior efficacy of weight freezing compared to traditional fully connected networks for EEG feature classification tasks. With its proven effectiveness, this innovative approach holds substantial promise for contributing to future strides in EEG decoding research.
Machine Learning,Signal Processing
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is: **How to improve the performance of artificial neural networks (ANN) in electroencephalogram (EEG) classification tasks by improving the structure and function of the fully - connected layer, especially in view of the characteristics of EEG signals with low signal - to - noise ratio and small amount of data**. Specifically, the paper introduced a new method named "Weight - Freezing", aiming to reduce the influence of certain neurons on the decision - making process by freezing some weights in the fully - connected layer. This method can not only achieve sparse connections, but also serve as an effective regularization means to prevent over - fitting, thereby improving the accuracy of EEG classification. ### Core problems and background of the paper 1. **Characteristics of EEG signals**: - EEG signals have low spatial resolution, low signal - to - noise ratio, as well as non - stationarity and inherent variability. - These characteristics make it very challenging to accurately identify the brain activities corresponding to EEG signals. 2. **Limitations of existing methods**: - Existing ANN models mainly focus on designing feature extraction networks, ignoring the optimization of the classifier part. - Due to its densely - connected characteristics, the fully - connected layer is prone to over - fitting when processing EEG data, especially in the case of a small amount of data. 3. **Proposal of Weight - Freezing**: - By freezing some weights in the fully - connected layer, Weight - Freezing can effectively suppress the influence of some input neurons on the decision results. - This method can not only achieve sparse connections, but also serve as a regularization means to improve classification performance. ### Experimental verification To verify the effectiveness of Weight - Freezing, the paper carried out the following experiments: - **Experimental setup**: - Three different ANN architectures (ConvNet, EEGNet, LMDA - Net) were used. - Three widely - recognized EEG datasets (BCI4 - 2A, BCI4 - 2B, High - Gamma) were tested. - **Experimental results**: - Weight - Freezing significantly improved the classification accuracy on all tested datasets. - Supplementary control experiments further analyzed the performance changes before and after Weight - Freezing and explored the influence of the threshold on the Weight - Freezing process. ### Main contributions 1. **For the first time, the influence of the classifier on EEG decoding performance was studied**, and the Weight - Freezing method was proposed. 2. **Sparse connections in the fully - connected layer were achieved**, reducing the number of learnable parameters, which is helpful for processing EEG signals with small amounts of data and low signal - to - noise ratios. 3. **The effectiveness of Weight - Freezing was verified on multiple classic decoding networks and public EEG datasets**, achieving state - of - the - art classification performance. In conclusion, this paper solves some key problems of existing ANN models in EEG classification tasks by introducing the Weight - Freezing method, providing new ideas and directions for future EEG decoding research.