Robin Manhaeve,Sebastijan Dumančić,Angelika Kimmig,Thomas Demeester,Luc De Raedt
Abstract:We introduce DeepProbLog, a probabilistic logic programming language that incorporates deep learning by means of neural predicates. We show how existing inference and learning techniques can be adapted for the new language. Our experiments demonstrate that DeepProbLog supports both symbolic and subsymbolic representations and inference, 1) program induction, 2) probabilistic (logic) programming, and 3) (deep) learning from examples. To the best of our knowledge, this work is the first to propose a framework where general-purpose neural networks and expressive probabilistic-logical modeling and reasoning are integrated in a way that exploits the full expressiveness and strengths of both worlds and can be trained end-to-end based on examples.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the challenge of combining low - level perception (handled by neural networks and deep learning) with high - level reasoning (achieved through logical and probabilistic representation and reasoning). Specifically, DeepProbLog aims to create a framework that combines general neural networks with expressive probabilistic - logical modeling and reasoning, thus making full use of the advantages of both and being able to perform end - to - end training based on examples.
### Main Problem Summary:
1. **Combining Low - level Perception and High - level Reasoning**:
- Low - level perception is usually handled by neural networks and deep learning.
- High - level reasoning is usually achieved using logical and probabilistic representation and reasoning.
2. **Integrating Symbolic and Sub - symbolic Representations**:
- Symbolic representation involves explicit rules and logical reasoning.
- Sub - symbolic representation involves continuous values and probability distributions, such as the weights in neural networks.
3. **Program Induction**:
- Automatically deriving a complete program or model from a small number of examples.
4. **Combining Probabilistic Programming and Deep Learning**:
- Combining the capabilities of probabilistic programming languages with the powerful representational capabilities of deep learning.
### Specific Contributions of DeepProbLog:
- **Supporting Symbolic and Sub - symbolic Representations**: DeepProbLog can handle both symbolic (such as logical rules) and sub - symbolic (such as neural network outputs) representations simultaneously.
- **Program Induction**: DeepProbLog can learn from a small number of examples and generate new logical rules.
- **Combining Probabilistic Programming and Deep Learning**: DeepProbLog extends ProbLog to enable it to include neural predicates, thus introducing deep learning into probabilistic - logical programming.
### Formula Representation:
The core idea of DeepProbLog is to interpret the output of a neural network as a probability and encapsulate it as a "neural predicate". For example, in the MNIST addition task, a neural predicate `digit/2` is defined to map digital images to the corresponding natural numbers:
\[
\text{addition}(X, Y, Z) \leftarrow \text{digit}(X, X_2), \text{digit}(Y, Y_2), Z \text{ is } X_2 + Y_2
\]
Here, `digit/2` is a neural predicate, and its output can be interpreted as a probability distribution. In this way, DeepProbLog can utilize the powerful representational capabilities of neural networks while maintaining the semantics of ProbLog.
### Training Process:
DeepProbLog trains neural network parameters through the back - propagation algorithm. Specifically, the loss function calculates the gradient of the probability distribution of the neural network output and passes it back to the neural network parameters through the chain rule. This process ensures that the entire model can be optimized end - to - end.
In summary, by introducing neural predicates, DeepProbLog successfully combines neural networks with probabilistic - logical programming, solves the problem of combining low - level perception and high - level reasoning, and demonstrates its superior performance on a variety of tasks.