Abstract:Recent work has seen the development of general purpose neural architectures that can be trained to perform tasks across diverse data modalities. General purpose models typically make few assumptions about the underlying data-structure and are known to perform well in the large-data regime. At the same time, there has been growing interest in modular neural architectures that represent the data using sparsely interacting modules. These models can be more robust out-of-distribution, computationally efficient, and capable of sample-efficient adaptation to new data. However, they tend to make domain-specific assumptions about the data, and present challenges in how module behavior (i.e., parameterization) and connectivity (i.e., their layout) can be jointly learned. In this work, we introduce a general purpose, yet modular neural architecture called Neural Attentive Circuits (NACs) that jointly learns the parameterization and a sparse connectivity of neural modules without using domain knowledge. NACs are best understood as the combination of two systems that are jointly trained end-to-end: one that determines the module configuration and the other that executes it on an input. We demonstrate qualitatively that NACs learn diverse and meaningful module configurations on the NLVR2 dataset without additional supervision. Quantitatively, we show that by incorporating modularity in this way, NACs improve upon a strong non-modular baseline in terms of low-shot adaptation on CIFAR and CUBs dataset by about 10%, and OOD robustness on Tiny ImageNet-R by about 2.5%. Further, we find that NACs can achieve an 8x speedup at inference time while losing less than 3% performance. Finally, we find NACs to yield competitive results on diverse data modalities spanning point-cloud classification, symbolic processing and text-classification from ASCII bytes, thereby confirming its general purpose nature.

Learn Basic Skills and Reuse: Modularized Adaptive Neural Architecture Search (MANAS)

MODULARNAS: TOWARDS MODULARIZED AND REUSABLE NEURAL ARCHITECTURE SEARCH

Manas: multi-agent neural architecture search

Advances in neural architecture search

Research on Autonomous Artificial Neural Network

Automated Search-Space Generation Neural Architecture Search

Deep Multimodal Neural Architecture Search

Architecture Self-Attention Mechanism: Nonlinear Optimization for Neural Architecture Search

Neural Attentive Circuits

Towards Less Constrained Macro-Neural Architecture Search

Cascaded Multi-task Adaptive Learning Based on Neural Architecture Search

AS-NAS: Adaptive Scalable Neural Architecture Search with Reinforced Evolutionary Algorithm for Deep Learning.

MathNAS: if Blocks Have a Role in Mathematical Architecture Design

AMEIR: Automatic Behavior Modeling, Interaction Exploration and MLP Investigation in the Recommender System

A Technical View on Neural Architecture Search

Understanding Architectures Learnt by Cell-based Neural Architecture Search

AutoIAS

A Semi-Supervised Assessor of Neural Architectures

Simplified Space Based Neural Architecture Search.

HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking

Task-Aware Neural Architecture Search