Insights into Cortical Oscillations Arising from Optogenetic Studies

V. Sohal

DOI: https://doi.org/10.1016/j.biopsych.2012.01.024

IF: 12.81

2012-06-15

Biological Psychiatry

Abstract:

What problem does this paper attempt to address?

$\alpha$ DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling

Bicheng Guo,Shuxuan Guo,Miaojing Shi,Peng Chen,Shibo He,Jiming Chen,Kaicheng Yu

DOI: https://doi.org/10.48550/arxiv.2211.10105

2022-01-01

Abstract: Differentiable architecture search (DARTS) has been a mainstream direction in automatic machine learning. Since the discovery that original DARTS will inevitably converge to poor architectures, recent works alleviate this by either designing rule-based architecture selection techniques or incorporating complex regularization techniques, abandoning the simplicity of the original DARTS that selects architectures based on the largest parametric value, namely $\alpha$. Moreover, we find that all the previous attempts only rely on classification labels, hence learning only single modal information and limiting the representation power of the shared network. To this end, we propose to additionally inject semantic information by formulating a patch recovery approach. Specifically, we exploit the recent trending masked image modeling and do not abandon the guidance from the downstream tasks during the search phase. Our method surpasses all previous DARTS variants and achieves state-of-the-art results on CIFAR-10, CIFAR-100, and ImageNet without complex manual-designed strategies.
Cyclic Differentiable Architecture Search

Hongyuan Yu,Houwen Peng,Yan Huang,Jianlong Fu,Hao Du,Liang Wang,Haibin Ling

DOI: https://doi.org/10.1109/tpami.2022.3153065

IF: 23.6

2022-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Differentiable ARchiTecture Search, i.e., DARTS, has drawn great attention in neural architecture search. It tries to find the optimal architecture in a shallow search network and then measures its performance in a deep evaluation network. The independent optimization of the search and evaluation networks, however, leaves a room for potential improvement by allowing interaction between the two networks. To address the problematic optimization issue, we propose new joint optimization objectives and a novel Cyclic Differentiable ARchiTecture Search framework, dubbed CDARTS. Considering the structure difference, CDARTS builds a cyclic feedback mechanism between the search and evaluation networks with introspective distillation. First, the search network generates an initial architecture for evaluation, and the weights of the evaluation network are optimized. Second, the architecture weights in the search network are further optimized by the label supervision in classification, as well as the regularization from the evaluation network through feature distillation. Repeating the above cycle results in a joint optimization of the search and evaluation networks and thus enables the evolution of the architecture to fit the final evaluation network. The experiments and analysis on CIFAR, ImageNet and NATS-Bench [95] demonstrate the effectiveness of the proposed approach over the state-of-the-art ones. Specifically, in the DARTS search space, we achieve 97.52% top-1 accuracy on CIFAR10 and 76.3% top-1 accuracy on ImageNet. In the chain-structured search space, we achieve 78.2% top-1 accuracy on ImageNet, which is 1.1% higher than EfficientNet-B0. Our code and models are publicly available at https://github.com/microsoft/Cream.

computer science, artificial intelligence,engineering, electrical & electronic
D-DARTS: Distributed Differentiable Architecture Search

Alexandre Heuillet,Hedi Tabia,Hichem Arioui,Kamal Youcef-Toumi

DOI: https://doi.org/10.48550/arXiv.2108.09306

2022-11-01

Abstract:Differentiable ARchiTecture Search (DARTS) is one of the most trending Neural Architecture Search (NAS) methods. It drastically reduces search cost by resorting to weight-sharing. However, it also dramatically reduces the search space, thus excluding potential promising architectures. In this article, we propose D-DARTS, a solution that addresses this problem by nesting neural networks at the cell level instead of using weight-sharing to produce more diversified and specialized architectures. Moreover, we introduce a novel algorithm that can derive deeper architectures from a few trained cells, increasing performance and saving computation time. In addition, we also present an alternative search space (DARTOpti) in which we optimize existing handcrafted architectures (e.g., ResNet) rather than starting from scratch. This approach is accompanied by a novel metric that measures the distance between architectures inside our custom search space. Our solution reaches competitive performance on multiple computer vision tasks. Code and pretrained models can be accessed at <a class="link-external link-https" href="https://github.com/aheuillet/D-DARTS" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Computer Vision and Pattern Recognition
PD-DARTS - Progressive Discretization Differentiable Architecture Search.

Yonggang Li,Yafeng Zhou,Yongtao Wang,Zhi Tang

DOI: https://doi.org/10.1007/978-3-030-59830-3_26

2020-01-01

Abstract:Architecture design is a crucial step for neural-network-based methods, and it requires years of experience and extensive work. Encouragingly, with recently proposed neural architecture search (NAS), the architecture design process could be automated. In particular, differentiable architecture search (DARTS) reduces the time cost of search to a couple of GPU days. However, due to the inconsistency between the architecture search and evaluation of DARTS, its performance has yet to be improved. We propose two strategies to narrow the search/evaluation gap: firstly, rectify the operation with the highest confidence; secondly, prune the operation with the lowest confidence iteratively. Experiments show that our method achieves 2.46%/2.48% (test error, Strategy 1 or 2) on CIFAR-10 and 16.48%/16.15% (test error, Strategy 1 or 2) on CIFAR-100 at a low cost of 11 or 8 (Strategy 1 or 2) GPU hours, and outperforms state-of-the-art algorithms.
Making Differentiable Architecture Search less local

Erik Bodin,Federico Tomasi,Zhenwen Dai

DOI: https://doi.org/10.48550/arXiv.2104.10450

IF: 5.414

2021-04-21

Machine Learning

Abstract:Neural architecture search (NAS) is a recent methodology for automating the design of neural network architectures. Differentiable neural architecture search (DARTS) is a promising NAS approach that dramatically increases search efficiency. However, it has been shown to suffer from performance collapse, where the search often leads to detrimental architectures. Many recent works try to address this issue of DARTS by identifying indicators for early stopping, regularising the search objective to reduce the dominance of some operations, or changing the parameterisation of the search problem. In this work, we hypothesise that performance collapses can arise from poor local optima around typical initial architectures and weights. We address this issue by developing a more global optimisation scheme that is able to better explore the space without changing the DARTS problem formulation. Our experiments show that our changes in the search algorithm allow the discovery of architectures with both better test performance and fewer parameters.
Single-DARTS: Towards Stable Architecture Search

Pengfei Hou,Ying Jin,Yukang Chen

DOI: https://doi.org/10.48550/arXiv.2108.08128

2021-08-18

Abstract:Differentiable architecture search (DARTS) marks a milestone in Neural Architecture Search (NAS), boasting simplicity and small search costs. However, DARTS still suffers from frequent performance collapse, which happens when some operations, such as skip connections, zeroes and poolings, dominate the architecture. In this paper, we are the first to point out that the phenomenon is attributed to bi-level optimization. We propose Single-DARTS which merely uses single-level optimization, updating network weights and architecture parameters simultaneously with the same data batch. Even single-level optimization has been previously attempted, no literature provides a systematic explanation on this essential point. Replacing the bi-level optimization, Single-DARTS obviously alleviates performance collapse as well as enhances the stability of architecture search. Experiment results show that Single-DARTS achieves state-of-the-art performance on mainstream search spaces. For instance, on NAS-Benchmark-201, the searched architectures are nearly optimal ones. We also validate that the single-level optimization framework is much more stable than the bi-level one. We hope that this simple yet effective method will give some insights on differential architecture search. The code is available at <a class="link-external link-https" href="https://github.com/PencilAndBike/Single-DARTS.git" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Machine Learning
Noisy Differentiable Architecture Search

Xiangxiang Chu,Bo Zhang

DOI: https://doi.org/10.48550/arXiv.2005.03566

2021-10-17

Abstract:Simplicity is the ultimate sophistication. Differentiable Architecture Search (DARTS) has now become one of the mainstream paradigms of neural architecture search. However, it largely suffers from the well-known performance collapse issue due to the aggregation of skip connections. It is thought to have overly benefited from the residual structure which accelerates the information flow. To weaken this impact, we propose to inject unbiased random noise to impede the flow. We name this novel approach NoisyDARTS. In effect, a network optimizer should perceive this difficulty at each training step and refrain from overshooting, especially on skip connections. In the long run, since we add no bias to the gradient in terms of expectation, it is still likely to converge to the right solution area. We also prove that the injected noise plays a role in smoothing the loss landscape, which makes the optimization easier. Our method features extreme simplicity and acts as a new strong baseline. We perform extensive experiments across various search spaces, datasets, and tasks, where we robustly achieve state-of-the-art results. Our code is available at <a class="link-external link-https" href="https://github.com/xiaomi-automl/NoisyDARTS" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Computer Vision and Pattern Recognition
Rethinking Bi-Level Optimization in Neural Architecture Search: A Gibbs Sampling Perspective.

Chao Xue,Xiaoxing Wang,Junchi Yan,Yonggang Hu,Xiaokang Yang,Kewei Sun

DOI: https://doi.org/10.1609/aaai.v35i12.17262

2021-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:One-Shot architecture search, which aims to explore all possible operations jointly based on a single model, has been an active direction of Neural Architecture Search (NAS). As a well-known one-shot solution, Differentiable Architecture Search (DARTS) performs continuous relaxation on the architecture's importance and results in a bi-level optimization problem. However, as many recent studies have shown, DARTS cannot always work robustly for new tasks, which is mainly due to the approximate solution of the bi-level optimization. In this paper, one-shot neural architecture search is addressed by adopting a directed probabilistic graphical model to represent the joint probability distribution over data and model. Then, neural architectures are searched for and optimized by Gibbs sampling. We rethink the bi-level optimization problem as the task of Gibbs sampling from the posterior distribution, which expresses the preferences for different models given the observed dataset. We evaluate our proposed NAS method -- GibbsNAS on the search space used in DARTS/ENAS and the search space of NAS-Bench-201. Experimental results on multiple search space show the efficacy and stability of our approach.
Differentiable Neural Architecture Search Via Proximal Iterations.

Quanming Yao,Jin Xu,Wei-Wei Tu,Zhanxing Zhu

2019-01-01

Abstract:Neural architecture search (NAS) recently attracts much research attention because of its ability to identify better architectures than handcrafted ones. However, many NAS methods, which optimize the search process in a discrete search space, need many GPU days for convergence. Recently, DARTS, which constructs a differentiable search space and then optimizes it by gradient descent, can obtain high-performance architecture and reduces the search time to several days. However, DARTS is still slow as it updates an ensemble of all operations and keeps only one after convergence. Besides, DARTS can converge to inferior architectures due to the strong correlation among operations. In this paper, we propose a new differentiable Neural Architecture Search method based on Proximal gradient descent (denoted as NASP). Different from DARTS, NASP reformulates the search process as an optimization problem with a constraint that only one operation is allowed to be updated during forward and backward propagation. Since the constraint is hard to deal with, we propose a new algorithm inspired by proximal iterations to solve it. Experiments on various tasks demonstrate that NASP can obtain high-performance architectures with 10 times of speedup on the computational time than DARTS.
Prioritized Architecture Sampling with Monto-Carlo Tree Search

Xiu Su,Tao Huang,Yanxi Li,Shan You,Fei Wang,Chen Qian,Changshui Zhang,Chang Xu

DOI: https://doi.org/10.1109/cvpr46437.2021.01082

2021-01-01

Abstract:One-shot neural architecture search (NAS) methods significantly reduce the search cost by considering the whole search space as one network, which only needs to be trained once. However, current methods select each operation independently without considering previous layers. Besides, the historical information obtained with huge computation costs is usually used only once and then discarded. In this paper, we introduce a sampling strategy based on Monte Carlo tree search (MCTS) with the search space modeled as a Monte Carlo tree (MCT), which captures the dependency among layers. Furthermore, intermediate results are stored in the MCT for future decisions and a better exploration-exploitation balance. Concretely, MCT is updated using the training loss as a reward to the architecture performance; for accurately evaluating the numerous nodes, we propose node communication and hierarchical node selection methods in the training and search stages, respectively, making better uses of the operation rewards and hierarchical information. Moreover, for a fair comparison of different NAS methods, we construct an open-source NAS benchmark of a macro search space evaluated on CIFAR-10, namely NAS-Bench-Macro. Extensive experiments on NAS-Bench-Macro and ImageNet demonstrate that our method significantly improves search efficiency and performance. For example, by only searching 20 architectures, our obtained architecture achieves 78.0% top-1 accuracy with 442M FLOPs on ImageNet. Code (Benchmark) is available at: https://github.com/xiusu/NAS-Bench-Macro.
Efficient Architecture Search via Bi-level Data Pruning

Chongjun Tu,Peng Ye,Weihao Lin,Hancheng Ye,Chong Yu,Tao Chen,Baopu Li,Wanli Ouyang

DOI: https://doi.org/10.48550/arXiv.2312.14200

2023-12-21

Abstract:Improving the efficiency of Neural Architecture Search (NAS) is a challenging but significant task that has received much attention. Previous works mainly adopted the Differentiable Architecture Search (DARTS) and improved its search strategies or modules to enhance search efficiency. Recently, some methods have started considering data reduction for speedup, but they are not tightly coupled with the architecture search process, resulting in sub-optimal performance. To this end, this work pioneers an exploration into the critical role of dataset characteristics for DARTS bi-level optimization, and then proposes a novel Bi-level Data Pruning (BDP) paradigm that targets the weights and architecture levels of DARTS to enhance efficiency from a data perspective. Specifically, we introduce a new progressive data pruning strategy that utilizes supernet prediction dynamics as the metric, to gradually prune unsuitable samples for DARTS during the search. An effective automatic class balance constraint is also integrated into BDP, to suppress potential class imbalances resulting from data-efficient algorithms. Comprehensive evaluations on the NAS-Bench-201 search space, DARTS search space, and MobileNet-like search space validate that BDP reduces search costs by over 50% while achieving superior performance when applied to baseline DARTS. Besides, we demonstrate that BDP can harmoniously integrate with advanced DARTS variants, like PC-DARTS and \b{eta}-DARTS, offering an approximately 2 times speedup with minimal performance compromises.

Computer Vision and Pattern Recognition
Partially-Connected Neural Architecture Search for Reduced Computational Redundancy

Yuhui Xu,Lingxi Xie,Wenrui Dai,Xiaopeng Zhang,Xin Chen,Guo-Jun Qi,Hongkai Xiong,Qi Tian

DOI: https://doi.org/10.1109/tpami.2021.3059510

IF: 23.6

2021-09-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Differentiable architecture search (DARTS) enables effective neural architecture search (NAS) using gradient descent, but suffers from high memory and computational costs. In this paper, we propose a novel approach, namely Partially-Connected DARTS (PC-DARTS), to achieve efficient and stable neural architecture search by reducing the channel and spatial redundancies of the super-network. In the channel level, partial channel connection is presented to randomly sample a small subset of channels for operation selection to accelerate the search process and suppress the over-fitting of the super-network. Side operation is introduced for bypassing (non-sampled) channels to guarantee the performance of searched architectures under extremely low sampling rates. In the spatial level, input features are down-sampled to eliminate spatial redundancy and enhance the efficiency of the mixed computation for operation selection. Furthermore, edge normalization is developed to maintain the consistency of edge selection based on channel sampling with the architectural parameters for edges. Theoretical analysis shows that partial channel connection and parameterized side operation are equivalent to regularizing the super-network on the weights and architectural parameters during bilevel optimization. Experimental results demonstrate that the proposed approach achieves higher search speed and training stability than DARTS. PC-DARTS obtains a top-1 error rate of 2.55 percent on CIFAR-10 with 0.07 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.1 percent on ImageNet (under the mobile setting) within 2.8 GPU-days.

computer science, artificial intelligence,engineering, electrical & electronic
$α$ DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling

Bicheng Guo,Shuxuan Guo,Miaojing Shi,Peng Chen,Shibo He,Jiming Chen,Kaicheng Yu

DOI: https://doi.org/10.48550/arXiv.2211.10105

2022-11-18

Abstract:Differentiable architecture search (DARTS) has been a mainstream direction in automatic machine learning. Since the discovery that original DARTS will inevitably converge to poor architectures, recent works alleviate this by either designing rule-based architecture selection techniques or incorporating complex regularization techniques, abandoning the simplicity of the original DARTS that selects architectures based on the largest parametric value, namely $\alpha$. Moreover, we find that all the previous attempts only rely on classification labels, hence learning only single modal information and limiting the representation power of the shared network. To this end, we propose to additionally inject semantic information by formulating a patch recovery approach. Specifically, we exploit the recent trending masked image modeling and do not abandon the guidance from the downstream tasks during the search phase. Our method surpasses all previous DARTS variants and achieves state-of-the-art results on CIFAR-10, CIFAR-100, and ImageNet without complex manual-designed strategies.

Computer Vision and Pattern Recognition
Improving Differentiable Architecture Search via self-distillation

Xunyu Zhu,Jian Li,Yong Liu,Weiping Wang

DOI: https://doi.org/10.1016/j.neunet.2023.08.062

IF: 7.8

2023-10-01

Neural Networks

Abstract:Differentiable Architecture Search (DARTS) is a simple yet efficient Neural Architecture Search (NAS) method. During the search stage, DARTS trains a supernet by jointly optimizing architecture parameters and network parameters. During the evaluation stage, DARTS discretizes the supernet to derive the optimal architecture based on architecture parameters. However, recent research has shown that during the training process, the supernet tends to converge towards sharp minima rather than flat minima. This is evidenced by the higher sharpness of the loss landscape of the supernet, which ultimately leads to a performance gap between the supernet and the optimal architecture. In this paper, we propose Self-Distillation Differentiable Neural Architecture Search (SD-DARTS) to alleviate the discretization gap. We utilize self-distillation to distill knowledge from previous steps of the supernet to guide its training in the current step, effectively reducing the sharpness of the supernet's loss and bridging the performance gap between the supernet and the optimal architecture. Furthermore, we introduce the concept of voting teachers, where multiple previous supernets are selected as teachers, and their output probabilities are aggregated through voting to obtain the final teacher prediction. Experimental results on real datasets demonstrate the advantages of our novel self-distillation-based NAS method compared to state-of-the-art alternatives.

computer science, artificial intelligence,neurosciences
EPC-DARTS: Efficient partial channel connection for differentiable architecture search

Zicheng Cai,Lei Chen,Hai-Lin Liu,Hai-lin Liu

DOI: https://doi.org/10.1016/j.neunet.2023.07.029

IF: 7.8

2023-07-01

Neural Networks

Abstract:With weight-sharing and continuous relaxation strategies, the differentiable architecture search (DARTS) proposes a fast and effective solution to perform neural network architecture search in various deep learning tasks. However, unresolved issues, such as the inefficient memory utilization, and the poor stability of the search architecture due to channels randomly selected, which has even caused performance collapses, are still perplexing researchers and practitioners. In this paper, a novel efficient channel attention mechanism based on partial channel connection for differentiable neural architecture search, termed EPC-DARTS, is proposed to address these two issues. Specifically, we design an efficient channel attention module, which is applied to capture cross-channel interactions and assign weight based on channel importance, to dramatically improve search efficiency and reduce memory occupation. Moreover, only partial channels with higher weights in the mixed calculation of operation are used through the efficient channel attention mechanism, and thus unstable network architectures obtained by the random selection operation can also be avoided in the proposed EPC-DARTS. Experimental results show that the proposed EPC-DARTS achieves remarkably competitive performance (CIFAR-10/CIFAR-100: a test accuracy rate of 97.60%/84.02%), compared to other state-of-the-art NAS methods using only 0.2 GPU-Days.

computer science, artificial intelligence,neurosciences
Latency-aware Neural Architecture Performance Predictor with Query-to-Tier Technique

Bicheng Guo,Lilin Xu,Tao Chen,Peng Ye,Shibo He,Haoyu Liu,Jiming Chen

DOI: https://doi.org/10.1109/tcsvt.2023.3287684

IF: 5.859

2024-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Neural Architecture Search (NAS) is a powerful tool for automating effective image and video processing DNN designing. The ranking of the accuracy has been advocated to design an efficient performance predictor for NAS. The previous contrastive method solves the ranking problem by comparing pairs of architectures and predicting their relative performance. However, it only focuses on the rankings between the two involved architectures and neglects the overall quality distributions of the search space, which may suffer generalization issues. On the contrary, we propose to let the performance predictor concentrate on the global quality level of specific architecture, and learn the tier embeddings of the whole search space automatically with learnable queries. The proposed method, dubbed as Neural Architecture Ranker with Query-to-Tier technique (NARQ2T), explores the quality tiers of the search space globally and classifies each individual to the tier they belong to. Thus, the predictor gains knowledge of the performance distributions of the search space which helps to generalize its ranking ability to the datasets more easily. Thanks to the encoder-decoder design, our method is able to predict the latency of the searched model without deteriorating the performance prediction. Meanwhile, the global quality distribution facilitates the search phase by directly sampling candidates according to the statistics of quality tiers, which is free of training a search algorithm, e.g., Reinforcement Learning or Evolutionary Algorithm, thus it simplifies the NAS pipeline and saves the computational overheads. The proposed NARQ2T achieves state-of-the-art performance on two widely used datasets for NAS research. Moreover, extensive experiments have validated the efficacy of the designed method.
Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Xiangxiang Chu,Tianbao Zhou,Bo Zhang,Jixiang Li

DOI: https://doi.org/10.48550/arXiv.1911.12126

2020-07-16

Abstract:Differentiable Architecture Search (DARTS) is now a widely disseminated weight-sharing neural architecture search method. However, it suffers from well-known performance collapse due to an inevitable aggregation of skip connections. In this paper, we first disclose that its root cause lies in an unfair advantage in exclusive competition. Through experiments, we show that if either of two conditions is broken, the collapse disappears. Thereby, we present a novel approach called Fair DARTS where the exclusive competition is relaxed to be collaborative. Specifically, we let each operation's architectural weight be independent of others. Yet there is still an important issue of discretization discrepancy. We then propose a zero-one loss to push architectural weights towards zero or one, which approximates an expected multi-hot solution. Our experiments are performed on two mainstream search spaces, and we derive new state-of-the-art results on CIFAR-10 and ImageNet. Our code is available on <a class="link-external link-https" href="https://github.com/xiaomi-automl/fairdarts" rel="external noopener nofollow">this https URL</a> .

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition
A neural network architecture optimizer based on DARTS and generative adversarial learning

Ting Zhang,Muhammad Waqas,Hao Shen,Zhaoying Liu,Xiangyu Zhang,Yujian Li,Zahid Halim,Sheng Chen

DOI: https://doi.org/10.1016/j.ins.2021.09.041

IF: 8.1

2021-12-01

Information Sciences

Abstract:Neural network architecture search automatically configures a set of network architectures according to the targeted rules. Thus, it relieves the human-dependent effort and repetitive resources consumption for designing neural network architectures and makes the task of finding the optimum network architecture with better performance much more accessible. Network architecture search methods based on differentiable architecture search (DARTS), however, introduces parameter redundancy. To address this issue, this work presents a novel method for optimizing network architectures that combines DARTS with generative adversarial learning (GAL). We first find the module structures utilizing the DARTS algorithm. Afterwards, the retrieved modules are stacked to derive the initial neural network architecture. Next, the GAL is used to prune some branches of the initial neural network, thereby obtaining the final neural network architecture. The proposed DARTS-GAL method re-optimizes the network architecture searched by DARTS to simplify the network connection and reduce network parameters without compromising network performance. Experimental results on benchmark datasets, i.e., Mixed National Institute of Standards and Technology (MNIST), FashionMNIST, Canadian Institute for Advanced Research10 (CIFAR10), Canadian Institute for Advanced Research100 (CIAFR100), Cats vs Dogs, and voiceprint recognition datasets, indicate that the test accuracies of the DARTS-GAL are higher than those of the DARTS in the majority of the cases. In particular, the proposed solution exhibits an improvement in accuracy by 7.35% on CIFAR10 compared with DARTS, attaining the state-of-the-art result of 99.60%. Additionally, the number of network parameters derived by the DARTS-GAL is significantly lower than that by the DARTS method, with a pruning rate of 62.3% at the highest case.

computer science, information systems
Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild

Xin Chen,Lingxi Xie,Jun Wu,Qi Tian

DOI: https://doi.org/10.1007/s11263-020-01396-x

IF: 13.369

2020-11-03

International Journal of Computer Vision

Abstract:With the rapid development of neural architecture search (NAS), researchers found powerful network architectures for a wide range of vision tasks. Like the manually designed counterparts, we desire the automatically searched architectures to have the ability of being freely transferred to different scenarios. This paper formally puts forward this problem, referred to as NAS in the wild, which explores the possibility of finding the optimal architecture in a proxy dataset and then deploying it to mostly unseen scenarios. We instantiate this setting using a currently popular algorithm named differentiable architecture search (DARTS), which often suffers unsatisfying performance while being transferred across different tasks. We argue that the accuracy drop originates from the formulation that uses a super-network for search but a sub-network for re-training. The different properties of these stages have resulted in a significant optimization gap, and consequently, the architectural parameters "over-fit" the super-network. To alleviate the gap, we present a progressive method that gradually increases the network depth during the search stage, which leads to the Progressive DARTS (P-DARTS) algorithm. With a reduced search cost (7 hours on a single GPU), P-DARTS achieves improved performance on both the proxy dataset (CIFAR10) and a few target problems (ImageNet classification, COCO detection and three ReID benchmarks). Our code is available at <span class="u-sans-serif">https://github.com/chenxin061/pdarts</span>.

computer science, artificial intelligence
OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strength

Le Yang,Ziwei Zheng,Yizeng Han,Shiji Song,Gao Huang,Fan Li

2024-09-22

Abstract:Differentiable architecture search (DARTS) has emerged as a promising technique for effective neural architecture search, and it mainly contains two steps to find the high-performance architecture: First, the DARTS supernet that consists of mixed operations will be optimized via gradient descent. Second, the final architecture will be built by the selected operations that contribute the most to the supernet. Although DARTS improves the efficiency of NAS, it suffers from the well-known degeneration issue which can lead to deteriorating architectures. Existing works mainly attribute the degeneration issue to the failure of its supernet optimization, while little attention has been paid to the selection method. In this paper, we cease to apply the widely-used magnitude-based selection method and propose a novel criterion based on operation strength that estimates the importance of an operation by its effect on the final loss. We show that the degeneration issue can be effectively addressed by using the proposed criterion without any modification of supernet optimization, indicating that the magnitude-based selection method can be a critical reason for the instability of DARTS. The experiments on NAS-Bench-201 and DARTS search spaces show the effectiveness of our method.

Artificial Intelligence

Insights into Cortical Oscillations Arising from Optogenetic Studies

$\alpha$ DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling

Cyclic Differentiable Architecture Search

D-DARTS: Distributed Differentiable Architecture Search

PD-DARTS - Progressive Discretization Differentiable Architecture Search.

Making Differentiable Architecture Search less local

Single-DARTS: Towards Stable Architecture Search

Noisy Differentiable Architecture Search

Rethinking Bi-Level Optimization in Neural Architecture Search: A Gibbs Sampling Perspective.

Differentiable Neural Architecture Search Via Proximal Iterations.

Prioritized Architecture Sampling with Monto-Carlo Tree Search

Efficient Architecture Search via Bi-level Data Pruning

Partially-Connected Neural Architecture Search for Reduced Computational Redundancy

$α$ DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling

Improving Differentiable Architecture Search via self-distillation

EPC-DARTS: Efficient partial channel connection for differentiable architecture search

Latency-aware Neural Architecture Performance Predictor with Query-to-Tier Technique

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

A neural network architecture optimizer based on DARTS and generative adversarial learning

Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild

OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strength