A Simple Generative Model of Logical Reasoning and Statistical Learning

Hiroyuki Kido

2023-05-19

Abstract:Statistical learning and logical reasoning are two major fields of AI expected to be unified for human-like machine intelligence. Most existing work considers how to combine existing logical and statistical systems. However, there is no theory of inference so far explaining how basic approaches to statistical learning and logical reasoning stem from a common principle. Inspired by the fact that much empirical work in neuroscience suggests Bayesian (or probabilistic generative) approaches to brain function including learning and reasoning, we here propose a simple Bayesian model of logical reasoning and statistical learning. The theory is statistically correct as it satisfies Kolmogorov's axioms, is consistent with both Fenstad's representation theorem and maximum likelihood estimation and performs exact Bayesian inference with a linear-time complexity. The theory is logically correct as it is a data-driven generalisation of uncertain reasoning from consistency, possibility, inconsistency and impossibility. The theory is correct in terms of machine learning as its solution to generation and prediction tasks on the MNIST dataset is not only empirically reasonable but also theoretically correct against the K nearest neighbour method. We simply model how data causes symbolic knowledge in terms of its satisfiability in formal logic. Symbolic reasoning emerges as a result of the process of going the causality forwards and backwards. The forward and backward processes correspond to an interpretation and inverse interpretation in formal logic, respectively. The inverse interpretation differentiates our work from the mainstream often referred to as inverse entailment, inverse deduction or inverse resolution. The perspective gives new insights into learning and reasoning towards human-like machine intelligence.

Artificial Intelligence

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is **how to unify the basic methods of logical reasoning and statistical learning from a common principle**. Specifically, the author proposes a simple Bayesian model to explain how the basic logical reasoning and statistical learning methods are derived from the same principle. This theory is not only statistically correct and satisfies the Kolmogorov axioms, but also logically correct and can perform uncertainty reasoning from a data - driven perspective. In addition, this theory also has applications in machine learning, providing a brand - new non - parametric all - nearest - neighbor classification method and its improved version to alleviate the over - fitting problem.

### Core contributions of the paper:
1. **Unifying logical reasoning and statistical learning**:
- A simplified Bayesian model is proposed to explain how the basic methods of logical reasoning and statistical learning originate from the same principle.
- The model shows how data leads to the generation of symbolic knowledge, and symbolic reasoning is generated through forward and backward causal processes.

2. **New method for the logical community**:
- A completely data - driven method for logical reasoning is provided, which has a linear time complexity and can handle cases of inconsistency and impossibility.

3. **New method for the machine - learning community**:
- A new non - parametric all - nearest - neighbor method and its improved version are proposed to alleviate the over - fitting problem.

### Theoretical basis:
- **Bayesian model**: This model is statistically correct, satisfies the Kolmogorov axioms, and is consistent with Fenstad's representation theorem and maximum - likelihood estimation.
- **Logical correctness**: This model is a method for reasoning about uncertainty from a data - driven perspective and can handle cases of consistency, possibility, inconsistency, and impossibility.
- **Machine - learning correctness**: In the generation and prediction tasks on the MNIST dataset, this model is not only experimentally reasonable but also theoretically verified.

### Specific formulas:
- **Generation process of probability distribution**:
\[
p(\alpha)=\sum_{n} p(\alpha|m_{n})p(m_{n})
\]
\[
p(\alpha)=\sum_{n} p(\alpha|m_{n})\sum_{k} p(m_{n}|d_{k})p(d_{k})
\]

- **Expansion of conditional probability**:
\[
p(\alpha|\Delta)=\sum_{n} p(\alpha|m_{n})p(m_{n}|\Delta)=\sum_{k} p(\alpha|d_{k})p(d_{k}|\Delta)
\]

- **Replacement of Bayes' theorem**:
\[
p(\alpha|\Delta)=\frac{\sum_{m} p(\Delta|m)p(\alpha|m)p(m)}{\sum_{m} p(\Delta|m)p(m)}
\]

### Experimental results:
- **Generation task**: On the MNIST dataset, this model can generate standard images from digits, and the results are both intuitively reasonable and statistically verified.
- **Prediction task**: On the MNIST dataset, this model can infer digits from digit images, and the results are not only experimentally reasonable but also theoretically verified.

Through these contributions, this paper provides a new perspective for unifying logical reasoning and statistical learning and also provides new classification methods for the field of machine learning.