Abstract:We revisit the classic signal-to-symbol barrier in light of the remarkable ability of deep neural networks to generate realistic synthetic data. DeepFakes and spoofing highlight the feebleness of the link between physical reality and its abstract representation, whether learned by a digital computer or a biological agent. Starting from a widely applicable definition of abstract concept, we show that standard feed-forward architectures cannot capture but trivial concepts, regardless of the number of weights and the amount of training data, despite being extremely effective classifiers. On the other hand, architectures that incorporate recursion can represent a significantly larger class of concepts, but may still be unable to learn them from a finite dataset. We qualitatively describe the class of concepts that can be "understood" by modern architectures trained with variants of stochastic gradient descent, using a (free energy) Lagrangian to measure information complexity. Even if a concept has been understood, however, a network has no means of communicating its understanding to an external agent, except through continuous interaction and validation. We then characterize physical objects as abstract concepts and use the previous analysis to show that physical objects can be encoded by finite architectures. However, to understand physical concepts, sensors must provide persistently exciting observations, for which the ability to control the data acquisition process is essential (active perception). The importance of control depends on the modality, benefiting visual more than acoustic or chemical perception. Finally, we conclude that binding physical entities to digital identities is possible in finite time with finite resources, solving in principle the signal-to-symbol barrier problem, but we highlight the need for continuous validation.

Do Deep Nets Really Need to be Deep?

Make Deep Networks Shallow Again

Efficient shallow learning as an alternative to deep learning

Shallow-Deep Networks: Understanding and Mitigating Network Overthinking

The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence

Deep Nets: What have they ever done for Vision?

From Deep to Shallow: Transformations of Deep Rectifier Networks

Generalization and Expressivity for Deep Nets

Are deep ResNets provably better than linear predictors?

Realizing Data Features by Deep Nets

Understanding deep learning (still) requires rethinking generalization

Efficient shallow learning mechanism as an alternative to deep learning

Deep Complex Networks

Learning Functions: When Is Deep Better Than Shallow

Are Efficient Deep Representations Learnable?

A note about why deep learning is deep: A discontinuous approximation perspective

Understanding deep learning requires rethinking generalization

Deep Speaker Verification: Do We Need End to End?

On the Learnability of Physical Concepts: Can a Neural Network Understand What's Real?

Learning Deep Architectures for AI

On the Depth of Deep Neural Networks: A Theoretical View