Employing Layerwised Unsupervised Learning to Lessen Data and Loss Requirements in Forward-Forward Algorithms

Taewook Hwang,Hyein Seo,Sangkeun Jung

2024-04-23

Abstract:Recent deep learning models such as ChatGPT utilizing the back-propagation algorithm have exhibited remarkable performance. However, the disparity between the biological brain processes and the back-propagation algorithm has been noted. The Forward-Forward algorithm, which trains deep learning models solely through the forward pass, has emerged to address this. Although the Forward-Forward algorithm cannot replace back-propagation due to limitations such as having to use special input and loss functions, it has the potential to be useful in special situations where back-propagation is difficult to use. To work around this limitation and verify usability, we propose an Unsupervised Forward-Forward algorithm. Using an unsupervised learning model enables training with usual loss functions and inputs without restriction. Through this approach, we lead to stable learning and enable versatile utilization across various datasets and tasks. From a usability perspective, given the characteristics of the Forward-Forward algorithm and the advantages of the proposed method, we anticipate its practical application even in scenarios such as federated learning, where deep learning layers need to be trained separately in physically distributed environments.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the following issues:

1. **Limitations of traditional Backpropagation (BP) algorithm**: Although the backpropagation algorithm performs well in deep learning, it significantly differs from the biological brain's processing, especially in terms of feedback mechanisms. Additionally, BP requires real-time connections across all layers for training, which is challenging in certain scenarios (e.g., federated learning).

2. **Limitations of Forward-Forward (FF) algorithm**: The FF algorithm simplifies the training process by using only forward passes, but it still requires specific inputs and loss functions, limiting its broad application. The FF method needs three types of inputs and special loss functions, making it unable to fully replace BP.

The paper proposes an unsupervised learning-based Forward-Forward algorithm (UFF), which overcomes the FF algorithm's need for special inputs and loss functions by employing unsupervised learning models (such as autoencoders, generative adversarial networks, etc.). This allows UFF to be trained using conventional data and loss functions. This approach not only enhances the model's flexibility but also enables more effective inter-layer training and information transfer in physically separated environments (e.g., federated learning scenarios). Experimental results show that UFF outperforms traditional BP methods in certain configurations (such as convolutional autoencoder FF) and exhibits more stable performance. However, UFF still lags behind BP in terms of training time and performance but has the potential to become an effective supplementary solution to BP, especially in scenarios where BP is difficult to apply.