Abstract:Neural architecture search (NAS) typically consists of three main steps: training a super-network, training and evaluating sampled deep neural networks (DNNs), and training the discovered DNN. Most of the existing efforts speed up some steps at the cost of a significant slowdown of other steps or sacrificing the support of non-differentiable search metrics. The unbalanced reduction in the time spent per step limits the total search time reduction, and the inability to support non-differentiable search metrics limits the performance of discovered DNNs. In this paper, we present NetAdaptV2 with three innovations to better balance the time spent for each step while supporting non-differentiable search metrics. First, we propose channel-level bypass connections that merge network depth and layer width into a single search dimension to reduce the time for training and evaluating sampled DNNs. Second, ordered dropout is proposed to train multiple DNNs in a single forward-backward pass to decrease the time for training a super-network. Third, we propose the multi-layer coordinate descent optimizer that considers the interplay of multiple layers in each iteration of optimization to improve the performance of discovered DNNs while supporting non-differentiable search metrics. With these innovations, NetAdaptV2 reduces the total search time by up to $5.8\times$ on ImageNet and $2.4\times$ on NYU Depth V2, respectively, and discovers DNNs with better accuracy-latency/accuracy-MAC trade-offs than state-of-the-art NAS works. Moreover, the discovered DNN outperforms NAS-discovered MobileNetV3 by 1.8% higher top-1 accuracy with the same latency. The project website is <a class="link-external link-http" href="http://netadapt.mit.edu" rel="external noopener nofollow">this http URL</a>.

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

ShuffleNeMt: modern lightweight convolutional neural network architecture

LA-ShuffleNet: A Strong Convolutional Neural Network for Edge Computing Devices

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

The Card Shuffling Hypotheses: Building a Time and Memory Efficient Graph Convolutional Network

SCGNet: Shifting and Cascaded Group Network

Model Parallelism Optimization for CNN FPGA Accelerator

Shift-based Primitives for Efficient Convolutional Neural Networks

HENet:A Highly Efficient Convolutional Neural Networks Optimized for Accuracy, Speed and Storage

AddressNet: Shift-Based Primitives for Efficient Convolutional Neural Networks.

A Reconfigurable Spatial Architecture for Energy-Efficient Inception Neural Networks

Neural Architecture Design for GPU-Efficient Networks

A Novel Low-Communication Energy-Efficient Reconfigurable CNN Acceleration Architecture

ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation

Effnet: An Efficient Structure for Convolutional Neural Networks

Exploring Hardware Friendly Bottleneck Architecture In Cnn For Embedded Computing Systems

ShuffleNet-inspired lightweight neural network design for automatic modulation classification methods in ubiquitous IoT cyber–physical systems

SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

Face Recognition Method Based on Lightweight Network SE-ShuffleNet V2

NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization