Abstract:The amount, size, and complexity of astronomical data-sets and databases are growing rapidly in the last decades, due to new technologies and dedicated survey telescopes. Besides dealing with poly-structured and complex data, sparse data has become a field of growing scientific interest. A specific field of Astroinformatics research is the estimation of redshifts of extra-galactic sources by using sparse photometric observations. Many techniques have been developed to produce those estimates with increasing precision. In recent years, models have been favored which instead of providing a point estimate only, are able to generate probabilistic density functions (PDFs) in order to characterize and quantify the uncertainties of their estimates. Crucial to the development of those models is a proper, mathematically principled way to evaluate and characterize their performances, based on scoring functions as well as on tools for assessing calibration. Still, in literature inappropriate methods are being used to express the quality of the estimates that are often not sufficient and can potentially generate misleading interpretations. In this work we summarize how to correctly evaluate errors and forecast quality when dealing with PDFs. We describe the use of the log-likelihood, the continuous ranked probability score (CRPS) and the probability integral transform (PIT) to characterize the calibration as well as the sharpness of predicted PDFs. We present what we achieved when using proper scoring rules to train deep neural networks as well as to evaluate the model estimates and how this work led from well calibrated redshift estimates to improvements in probabilistic weather forecasting. The presented work is an example of interdisciplinarity in data-science and illustrates how methods can help to bridge gaps between different fields of application.

Why Machine Learning Models Systematically Underestimate Extreme Values

Systematic Bias in Sample Inference and its Effect on Machine Learning

A Systematic Bias of Machine Learning Regression Models and Its Correction: an Application to Imaging-based Brain Age Prediction

Regression for Astronomical Data with Realistic Distributions, Errors and Non-linearity

Measurement errors and scaling relations in astrophysics: a review

Negative impact of heavy-tailed uncertainty and error distributions on the reliability of calibration statistics for machine learning regression tasks

Machine Learning LSST 3x2pt analyses -- forecasting the impact of systematics on cosmological constraints using neural networks

Large-scale power loss in ground-based CMB mapmaking

Reconciling modern machine learning practice and the bias-variance trade-off

Validity Concerns of Using Machine Learning in Management Research

Unraveling overoptimism and publication bias in ML-driven science

Exploring galactic properties with machine learning Predicting star formation, stellar mass, and metallicity from photometric data

From Photometric Redshifts to Improved Weather Forecasts: machine learning and proper scoring rules as a basis for interdisciplinary work

Distance-based Analysis of Machine Learning Prediction Reliability for Datasets in Materials Science and Other Fields

Analyzing Astronomical Data with Machine Learning Techniques

Exploring galactic properties with machine learning

AstroMLab 1: Who Wins Astronomy Jeopardy!?

Machine Learning in Astronomy: a practical overview

De-biasing "bias" measurement

Biased Moments of Undersampled Sources

Conformal Prediction for Astronomy Data with Measurement Error