Abstract:Within one decade, Deep Learning overtook the dominating solution methods of countless problems of artificial intelligence. ``Deep'' refers to the deep architectures with operations in manifolds of which there are no immediate observations. For these deep architectures some kind of structure is pre-defined -- but what is this structure? With a formal definition for structures of neural networks, neural architecture search problems and solution methods can be formulated under a common framework. Both practical and theoretical questions arise from closing the gap between applied neural architecture search and learning theory. Does structure make a difference or can it be chosen arbitrarily? This work is concerned with deep structures of artificial neural networks and examines automatic construction methods under empirical principles to shed light on to the so called ``black-box models''. Our contributions include a formulation of graph-induced neural networks that is used to pose optimisation problems for neural architecture. We analyse structural properties for different neural network objectives such as correctness, robustness or energy consumption and discuss how structure affects them. Selected automation methods for neural architecture optimisation problems are discussed and empirically analysed. With the insights gained from formalising graph-induced neural networks, analysing structural properties and comparing the applicability of neural architecture search methods qualitatively and quantitatively we advance these methods in two ways. First, new predictive models are presented for replacing computationally expensive evaluation schemes, and second, new generative models for informed sampling during neural architecture search are analysed and discussed.

Structures of Neural Network Effective Theories

Structures of neural network effective theories

Neural Network Field Theories: Non-Gaussianity, Actions, and Locality

Linking Network and Neuron-level Correlations by Renormalized Field Theory

Deep Neural Networks via Complex Network Theory: a Perspective

Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristics

Statistical Field Theory and Neural Structures Dynamics I: Action Functionals, Background States and External Perturbations

Learning and Structure of Neuronal Networks

Canonical Cortical Field Theories

Statistical Field Theory and Neural Structures Dynamics II: Signals Propagation, Interferences, Bound States

Statistical Field Theory and Neural Structures Dynamics III: Effective Action for Connectivities, Interactions and Emerging Collective States

A Neural Circuit Theory for Neuroinformatics and Brain-Machine Interactions

Connectivity structure and dynamics of nonlinear recurrent neural networks

GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory

Nonequilibrium landscape theory of neural networks

Tensor formalism for predicting synaptic connections with ensemble modeling or optimization

Statistical Field Theory and Neural Structures Dynamics IV: Field-Theoretic Formalism for Interacting Collective States

Criticality & Deep Learning I: Generally Weighted Nets

Dynamic effective connectivity of inter-areal brain circuits

Structure of Artificial Neural Networks -- Empirical Investigations

A General Description of Criticality in Neural Network Models