Abstract:As an emerging and popular technique for boosting CNNs, structural reparameterization (SR) decouples the training and inference structures to alter the training dynamics and achieve cost-free improvement of a given network. Existing SR methods often prioritize network expressiveness enhancement but have yet to investigate approaches to mitigate significant bias and non-robustness of model prediction due to over-reliance on training data distribution and image noise. To this end, inspired by the effective strength of implicit regularization on the problem, this paper introduces an extra balanced implicit regularization mechanism into SR techniques to enhance the generalization of a given network for the first time. Specifically, we propose a novel SR module named DR-Block, which is used to complicate each convolutional layer of a given CNN during training. It draws on the advantages of deep matrix factorization with the regularization effect and further improves singular value dynamics by introducing batch normalization and dense connections to alleviate network degradation. At inference time, DR-Block can be equivalently reparameterized back into a single convolution for deployment. Furthermore, we empirically demonstrate the role of each design in DR-Block and explicitly reveal its inherent mechanism, which lies in enhancing the movement of large singular values while countering the attenuation of small ones. This helps enhance the interpretability of SR techniques. Experiments illustrate that DR-Block is an impressive alternative for a regular convolution layer of any structure and outperforms the existing SR methods in improving mainstream network architectures on various visual tasks. The code is available at https://github.com/qyan0131/DRBlock.

CMB: A Novel Structural Re-parameterization Block Without Extra Training Parameters.

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

DR-Block: Convolutional Dense Reparameterization for CNN Generalization Free Improvement

Expansion-Squeeze- Block: Linear Over-parameterization with Shortcut Connections to Train Compact Convolutional Networks

Generalized Enhanced Convolution Block for Super-Resolution Networks

RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization

ACNet: Strengthening the Kernel Skeletons for Powerful CNN Via Asymmetric Convolution Blocks

IC Networks: Remodeling the Basic Unit for Convolutional Neural Networks

Block-Wise Partner Learning for Model Compression.

IC-Network: Efficient Structure for Convolutional Neural Networks

Receptive Field Refinement for Convolutional Neural Networks Reliably Improves Predictive Performance

RepNeXt: A Fast Multi-Scale CNN using Structural Reparameterization

Compact Cross-Reparam Convolution Network for Efficient Image Super-resolution.

RDPNet: a Single-Path Lightweight CNN with Re-Parameterization for CPU-type Edge Devices

Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy

Efficient Network Construction Through Structural Plasticity

Reparameterization through Spatial Gradient Scaling

CNN LEGO: Disassembling and Assembling Convolutional Neural Network

A very lightweight image super-resolution network

REPNAS: SEARCHING FOR EFFICIENT RE-PARAMETERIZING BLOCKS

Column Row Convolutional Neural Network: Reducing Parameters for Efficient Image Processing