Abstract:This paper considers similarities between statistical physics and Bayes inference through the Bayesian linear regression model. Some similarities have been discussed previously, such as the analogy between the marginal likelihood in Bayes inference and the partition function in statistical mechanics. In particular, this paper considers the proposal to associate discrete sample size with inverse temperature [C. H. LaMont and P. A. Wiggins, Phys. Rev. E 99 , 052140 (2019)]. The previous study suggested that incorporating this similarity motivates the derivation of analogs of thermodynamic functions such as energy and entropy. The study also anticipated that those analogous functions have potential to describe Bayes estimation from physical points of view and to provide physical insights into mechanisms of estimation. This paper incorporates a macroscopic perspective as an asymptotics similar to the thermodynamic limit into the previous suggestion. Its motivation stems from the statistical mechanical concept of deriving thermodynamic functions that characterize macroscopic properties of macroscopic systems. This incorporation not only allows analogs of macroscopic thermodynamic functions to be considered but also suggests a candidate for an analog of inverse temperature with continuity, which is partly consistent with the previous proposal to associate the discrete sample size with inverse temperature. On the basis of this suggestion, we analyze analogs of macroscopic thermodynamic functions for a Bayesian linear regression model which is the basis of various machine learning models. We further investigate, through the behavior of these functions, how Bayes estimation is described from the perspective of physics and what kind of physical insight is obtained. As a result, the estimation of regression coefficients, which is the primary task of regression, appears to be described by the physical picture of balance between decreasing energy and increasing entropy as in equilibrium states of thermodynamic systems. More specifically we observe the physical view of Bayes inference as follows: the estimation succeeds where the effect of decreasing energy is dominant at low temperature. On the other hand, the estimation fails where the effect of increasing entropy is dominant at high temperature. https://doi.org/10.1103/PhysRevE.110.034118 ©2024 American Physical Society

Hilbert's Sixth Problem: Descriptive Statistics as New Foundations for Probability

The probabilistic world

Theory of Probability. A Historical Essay

Many Faces of Entropy or Bayesian Statistical Mechanics

The foundations of statistical physics: entropy, irreversibility, and inference

Bayesian approach to superstatistics

THE 4-DIMENSIONAL STATISTICAL PHYSICS IN A RELATIONAL PARADIGM

Statistical physical view of statistical inference in Bayesian linear regression model

Probabilistic Knowledge as Objective Knowledge in Quantum Mechanics: Potential Powers Instead of Actual Properties

Extended Probabilities: Mathematical Foundations

The causal foundations of applied probability and statistics

A Novel Approach to Probability

Basic Principles and Applications of Probability Theory

On the foundations of statistical mechanics

Causality from the Point of View of Statistics

The basic paradoxes of statistical classical physics and the quantum mechanics

Fundamental Principle for Quantum Theory

History and outlook of statistical physics

Statistical Patterns in the Equations of Physics and the Emergence of a Meta-Law of Nature

Quantum mechanics as a theory of probability