Abstract:As a result of its low search cost, Differentiable Architecture Search (DARTS) has recently received a lot of interest. Nowadays, most methods based on DARTS only focus on improving a single indicator (e.g., accuracy), making the search process more inclined to complex networks with more robust representational capabilities. Hence, the architectures searched by these methods tend to own high latency, leading to DARTS in some low-latency scenarios or edges with limited computing power, and on-device deployment becomes difficult. Intuitively, searching for an optimal latency-friendly architecture for devices with different computing power is an excellent solution to help the DARTS algorithm better deploy, but it is also a challenge. To deal with this challenge, we propose a Hardware Latency-Aware Differentiable Search (HL-DARTS) algorithm. This algorithm designs a multi-layer regression network that uses the soft attention mechanism to predict the latency on the corresponding hardware devices, thus adding a differentiable latency loss term based on the DARTS algorithm. We further propose an adaptive constraint amplitude—a mechanism for balancing accuracy and latency while searching for a latency-friendly architecture for a given hardware device. We conduct experiments in the CIFAR-10 dataset. The latency and number of parameters of the architectures searched by HL-DARTS are significantly decreased with nearly the same accuracy as the baseline DARTS algorithm. We also conduct ablation experiments on different datasets (CIFAR100, MIO-TCD, ImageNet) and different hardware devices (GPU, Intel-CPU, AMD-CPU). The experimental results show that HL-DARTS can find the ideal architecture for different hardware devices and that this architecture is also broadly applicable to various datasets.

Hardware Latency-Aware Differential Architecture Search: Search for Latency-Friendly Architectures on Different Hardware

Latency-Aware Differentiable Neural Architecture Search

PD-DARTS - Progressive Discretization Differentiable Architecture Search.

$\alpha$ DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling

ℓ-Darts: Light-weight Differentiable Architecture Search with Robustness Enhancement Strategy

Cyclic Differentiable Architecture Search

Efficient Search for Efficient Architecture.

Sparse Gate for Differentiable Architecture Search

Enhancing the Robustness, Efficiency, and Diversity of Differentiable Architecture Search

D-DARTS: Distributed Differentiable Architecture Search

EPC-DARTS: Efficient partial channel connection for differentiable architecture search

AD-DARTS - Adaptive Dropout for Differentiable Architecture Search.

PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search

PO-DARTS: Post-optimizing the Architectures Searched by Differentiable Architecture Search Algorithms

Operation-level Progressive Differentiable Architecture Search

Layered Feature Representation for Differentiable Architecture Search

Single-level Optimization For Differential Architecture Search

Regularized Differentiable Architecture Search.

Hardware-Aware Transformable Architecture Search with Efficient Search Space.

MC-PDARTS: Multi-Cell Progressive Differentiable Architecture Search.

Making Differentiable Architecture Search less local