Abstract:The empirical risk minimization (ERM) problem with relative entropy regularization (ERM-RER) is investigated under the assumption that the reference measure is a $\sigma$-finite measure, and not necessarily a probability measure. Under this assumption, which leads to a generalization of the ERM-RER problem allowing a larger degree of flexibility for incorporating prior knowledge, numerous relevant properties are stated. Among these properties, the solution to this problem, if it exists, is shown to be a unique probability measure, mutually absolutely continuous with the reference measure. Such a solution exhibits a probably-approximately-correct guarantee for the ERM problem independently of whether the latter possesses a solution. For a fixed dataset and under a specific condition, the empirical risk is shown to be a sub-Gaussian random variable when the models are sampled from the solution to the ERM-RER problem. The generalization capabilities of the solution to the ERM-RER problem (the Gibbs algorithm) are studied via the sensitivity of the expected empirical risk to deviations from such a solution towards alternative probability measures. Finally, an interesting connection between sensitivity, generalization error, and lautum information is established.

Conditional supremum in Riesz spaces and applications

Quasi-sure essential supremum and applications to finance

Generating functions in Riesz spaces

Conditional Expectations in Banach spaces with RNP

Conditional probability in Renyi spaces

On optimal transport of matrix-valued measures

Conditional indicators

Exponential inequalities in probability spaces revisited

A note on maximal conditional entropy on Lebesgue spaces

Riesz spaces of signed charges on semi-rings

Conditional expectations of random holomorphic fields on Riemann surfaces

A Hahn-Jordan decomposition and Riesz-Frechet representation theorem in Riesz spaces

Conditional Carleson measures and related operators on Bergman spaces

On the new properties of conditional expectations with applications in finance

Empirical Risk Minimization with Relative Entropy Regularization

Integral-Type Representations for the Subdifferential of Suprema

Fractal surfaces in Lebesgue spaces with respect to fractal measures and associated fractal operators

Graphical models for infinite measures with applications to extremes

Sparse Inverse Problems Over Measures: Equivalence of the Conditional Gradient and Exchange Methods

Monge-Kantorovich superquantiles and expected shortfalls with applications to multivariate risk measurements