VanillaNet: the Power of Minimalism in Deep Learning

Hanting Chen,Yunhe Wang,Jianyuan Guo,Dacheng Tao

2023-05-23

Abstract:At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at <a class="link-external link-https" href="https://github.com/huawei-noah/VanillaNet" rel="external noopener nofollow">this https URL</a> and <a class="link-external link-https" href="https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the complexity and optimization challenges present in current deep learning models. Specifically:

- **Simplifying Network Design**: Current deep learning models (such as ResNet, ViT, etc.) are powerful in performance but their complex design poses difficulties for deployment, especially in resource-constrained environments. Therefore, the paper proposes a new architecture called VanillaNet, which abandons complex layers, residual connections, and self-attention mechanisms to achieve a simple yet efficient model.
- **Enhancing Shallow Network Performance**: Traditional shallow networks (such as AlexNet and VGGNet) have been outperformed by deep networks in terms of performance. VanillaNet significantly improves the performance of shallow networks by introducing a series of techniques (such as deep training strategies and serial activation functions), enabling them to compete with existing top models in computer vision tasks.
- **Efficient Inference**: The development of modern AI chips has endowed GPUs with powerful computational capabilities. Therefore, the paper emphasizes the need to consider the requirements of practical application scenarios in network design. VanillaNet not only matches the performance of existing models but also has a significant advantage in inference speed, especially for real-time processing on GPUs.

In summary, VanillaNet aims to simplify network design, enhance the performance of shallow networks, and optimize their deployment efficiency in practical applications.