Increased accuracy of the two-point correlation function at no extra cost

Martin Kerscher

2024-01-11

Abstract:Using the pair-count implementaion from the Corrfunc package we show that with a low discrepency sequence we can calculate the two-point correlation function more accurately than with random points at no extra computational cost.

Cosmology and Nongalactic Astrophysics,Instrumentation and Methods for Astrophysics

What problem does this paper attempt to address?

Pair Counting without Binning -- A New Approach to Correlation Functions in Clustering Statistics

Shiyu Yue,Longlong Feng,Wenjie Ju,Jun Pan,Zhiqi Huang,Feng Fang,Zhuoyang Li,Yan-Chuan Cai,Weishan Zhu

2024-08-29

Abstract:This paper presents a novel perspective on correlation functions in the clustering analysis of the large-scale structure of the universe. We first recognise that pair counting in bins of radial separation is equivalent to evaluating counts-in-cells (CIC), which can be modelled using a filtered density field with a binning-window function. This insight leads to an in situ expression for the two-point correlation function (2PCF). Essentially, the core idea underlying our method is to introduce a window function to define the binning scheme, enabling pair-counting without binning. This approach develops a concept of generalised 2PCF, which extends beyond conventional discrete pair counting by accommodating non-sharp-edged window functions. To extend this framework to N-point correlation functions (NPCF) using current optimal edge-corrected estimators, we developed a binning scheme independent of the specific parameterisation of polyhedral configurations. In particular, we demonstrate a fast algorithm for the three-point correlation function (3PCF), where triplet counting is accomplished by assigning either a spherical tophat or a Gaussian filter to each vertex of triangles. Additionally, we derive analytical expressions for the 3PCF using a multipole expansion in Legendre polynomials, accounting for filtered field (binning) corrections. Numerical tests using several suites of N-body simulation samples show that our approach aligns remarkably well with the theoretical predictions. Our method provides an exact solution for quantifying binning effects in practical measurements and offers a high-speed algorithm, enabling high-order clustering analysis in extremely large datasets from ongoing and upcoming surveys such as Euclid, LSST, and DESI.

Cosmology and Nongalactic Astrophysics
The 2-point correlation function covariance with fewer mocks

Svyatoslav Trusov,Pauline Zarrouk,Shaun Cole,Peder Norberg,Cheng Zhao,Jessica Nicole Aguilar,Steven Ahlen,David Brooks,Axel de la Macorra,Peder Doel,Andreu Font-Ribera,Klaus Honscheid,Theodore Kisner,Martin Landriau,Christophe Magneville,Ramon Miquel,Jundan Nie,Claire Poppett,Michael Schubnell,Gregory Tarlé,Zhimin Zhou

DOI: https://doi.org/10.1093/mnras/stad3710

2024-01-09

Abstract:We present an approach for accurate estimation of the covariance of 2-point correlation functions that requires fewer mocks than the standard mock-based covariance. This can be achieved by dividing a set of mocks into jackknife regions and fitting the correction term first introduced in Mohammad & Percival (2022), such that the mean of the jackknife covariances corresponds to the one from the mocks. This extends the model beyond the shot-noise limited regime, allowing it to be used for denser samples of galaxies. We test the performance of our fitted jackknife approach, both in terms of accuracy and precision, using lognormal mocks with varying densities and approximate EZmocks mimicking the DESI LRG and ELG samples in the redshift range of z = [0.8, 1.2]. We find that the Mohammad-Percival correction produces a bias in the 2-point correlation function covariance matrix that grows with number density and that our fitted jackknife approach does not. We also study the effect of the covariance on the uncertainty of cosmological parameters by performing a full-shape analysis. We find that our fitted jackknife approach based on 25 mocks is able to recover unbiased and as precise cosmological parameters as the ones obtained from a covariance matrix based on 1000 or 1500 mocks, while the Mohammad-Percival correction produces uncertainties that are twice as large. The number of mocks required to obtain an accurate estimation of the covariance for 2-point correlation function is therefore reduced by a factor of 40-60.

Cosmology and Nongalactic Astrophysics
corr2D - Implementation of Two-Dimensional Correlation Analysis in R

Robert Geitner,Robby Fritzsch,Jürgen Popp,Thomas W. Bocklitz

DOI: https://doi.org/10.18637/jss.v090.i03

2018-08-02

Abstract:In the package corr2D two-dimensional correlation analysis is implemented in R. This paper describes how two-dimensional correlation analysis is done in the package and how the mathematical equations are translated into R code. The paper features a simple tutorial with executable code for beginners, insight into at the calculations done before the correlation analysis, a detailed look at the parallelization of the fast Fourier transformation based correlation analysis and a speed test of the calculation. The package corr2D offers the possibility to preprocess, correlate and postprocess spectroscopic data using exclusively the R language. Thus, corr2D is a welcome addition to the toolbox of spectroscopists and makes two-dimensional correlation analysis more accessible and transparent.

Computation
Improved two-point correlation function estimates using glass-like distributions as a reference sample

Federico Dávila-Kurbán,Ariel G Sánchez,Marcelo Lares,Andrés N Ruiz

DOI: https://doi.org/10.1093/mnras/stab1622

IF: 4.8

2021-06-14

Monthly Notices of the Royal Astronomical Society

Abstract:ABSTRACT All estimators of the two-point correlation function are based on a random catalogue, a set of points with no intrinsic clustering following the selection function of a survey. High-accuracy estimates require the use of large random catalogues, which imply a high computational cost. We propose to replace the standard random catalogues by glass-like point distributions or glass catalogues whose power spectrum P(k) ∝ k4 exhibits significantly less power on scales larger than the mean interparticle separation than a Poisson distribution with the same number of points. We show that these distributions can be obtained by iteratively applying the technique of Zeldovich reconstruction commonly used in studies of baryon acoustic oscillations (BAO). We provide a modified version of the widely used Landy–Szalay estimator of the correlation function adapted to the use of glass catalogues and compare its performance with the results obtained using random samples. Our results show that glass-like samples do not add any bias with respect to the results obtained using Poisson distributions. On scales larger than the mean interparticle separation of the glass catalogues, the modified estimator leads to a significant reduction of the variance of the Legendre multipoles ξl(s) with respect to the standard Landy–Szalay results with the same number of points. The size of the glass catalogue required to achieve a given accuracy in the correlation function is significantly smaller than when using random samples. Their use could help us to drastically reduce the computational cost of configuration-space clustering analysis of future surveys while maintaining high-accuracy requirements.

astronomy & astrophysics
A Set of Efficient Methods to Generate High-Dimensional Binary Data with Specified Correlation Structures

Wei Jiang,Shuang Song,Lin Hou,Hongyu Zhao

DOI: https://doi.org/10.1080/00031305.2020.1816213

2020-01-01

The American Statistician

Abstract:High-dimensional correlated binary data arise in many areas, such as observed genetic variations in biomedical research. Data simulation can help researchers evaluate efficiency and explore properties of different computational and statistical methods. Also, some statistical methods, such as Monte Carlo methods, rely on data simulation. Lunn and Davies proposed linear time complexity methods to generate correlated binary variables with three common correlation structures. However, it is infeasible to specify unequal probabilities in their methods. In this article, we introduce several computationally efficient algorithms that generate high-dimensional binary data with specified correlation structures and unequal probabilities. Our algorithms have linear time complexity with respect to the dimension for three commonly studied correlation structures, namely exchangeable, decaying-product and K-dependent correlation structures. In addition, we extend our algorithms to generate binary data of specified nonnegative correlation matrices satisfying the validity condition with quadratic time complexity. We provide an R package, CorBin, to implement our simulation methods. Compared to the existing packages for binary data generation, the time cost to generate a 100-dimensional binary vector with the common correlation structures and general correlation matrices can be reduced up to 105 folds and 103 folds, respectively, and the efficiency can be further improved with the increase of dimensions. The R package CorBin is available on CRAN at https://cran.r-project.org/.
A new method for the robust characterisation of pairwise statistical dependency between point processes

Antoine Messager,Nicos Georgiou,Luc Berthouze

DOI: https://doi.org/10.48550/arXiv.1904.04813

2019-07-31

Abstract:The robust detection of statistical dependencies between the components of a complex system is a key step in gaining a network-based understanding of the system. Because of their simplicity and low computation cost, pairwise statistics are commonly used in a variety of fields. Those approaches, however, typically suffer from one or more limitations such as lack of confidence intervals requiring reliance on surrogate data, sensitivity to binning, sparsity of the signals, or short duration of the records. In this paper we develop a method for assessing pairwise dependencies in point processes that overcomes these challenges. Given two point processes $X$ and $Y$ each emitting a given number of events $m$ and $n$ in a fixed period of time $T$, we derive exact analytical expressions for the expected value and standard deviation of the number of pairs events $X_i,Y_j$ separated by a delay of less than $\tau$ one should expect to observe if $X$ and $Y$ were i.i.d. uniform random variables. We prove that this statistic is normally distributed in the limit of large $T$, which enables the definition of a Z-score characterising the likelihood of the observed number of coincident events happening by chance. We numerically confirm the analytical results and show that the property of normality is robust in a wide range of experimental conditions. We then experimentally demonstrate the predictive power of the method using a noisy version of the common shock model. Our results show that our approach has excellent behaviour even in scenarios with low event density and/or when the recordings are short.

Statistics Theory
Calculating pair-correlations from random particle configurations

Aris Karnezis,Art L. Gower

2024-05-30

Abstract:Particle pair-correlations are broadly used to describe particle distributions in chemistry, physics, and material science. Many theoretical methods require the pair-correlation to predict material properties such as fluid flow, thermal properties, or wave propagation. In all these applications it is either important to calculate a pair-correlation from specific particle configurations, or vice-versa: determine the likely particle configurations from a pair-correlation which is needed to fabricate a particulate material. Most available methods to calculate the pair-correlation from a particle configuration require that the configuration be very large to avoid effects from the boundary. Here we show how to avoid boundary effects even for small particle configurations. Having small particle configurations leads to far more efficient numerical methods. We also demonstrate how to use techniques from smooth nonlinear optimisation to quickly recover a particle configuration from a pair-correlation.

Soft Condensed Matter
An Improved Estimator for the Correlation Function of 2D Nonlinear Sigma Models

Martin Hasenbusch

DOI: https://doi.org/10.1016/0920-5632%2895%2900375-J

1994-08-25

Abstract:I present a new improved estimator for the correlation function of 2D nonlinear sigma models. Numerical tests for the 2D XY model and the 2D O(3)-invariant vector model were performed. For small physical volume, i.e. a lattice size small compared to the to the bulk correlation length, a reduction of the statistical error of the finite system correlation length by a factor of up to 30 compared to the cluster-improved estimator was observed. This improvement allows for a very accurate determination of the running coupling proposed by M. L"uscher et al. for 2D O(N)-invariant vector models.

High Energy Physics - Lattice
On the smallest gap in a sequence with Poisson pair correlations

Daniel Altman,Zachary Chase

DOI: https://doi.org/10.48550/arXiv.2210.17466

2022-11-01

Abstract:We prove that any increasing sequence of real numbers with average gap $1$ and Poisson pair correlations has some gap that is at least $3/2+10^{-9}$. This improves upon a result of Aistleitner, Blomer, and Radziwill.

Combinatorics,Number Theory
Algorithm to Produce a Density Field with Given Two, Three, and Four-Point Correlation Functions

Zachary Slepian

2024-07-13

Abstract:Here we show how to produce a 3D density field with a given set of higher-order correlation functions. Our algorithm enables producing any desired two-point, three-point, and four-point functions, including odd-parity for the latter. We note that this algorithm produces the desired correlations about a set of ``primary'' points, matched to how the spherical-harmonic-based algorithms ENCORE and CADENZA measure them. These ``primary points'' must be used as those around which the correlation functions are measured. We also generalize the algorithm to i) $N$-point correlations with $N>4$, ii) dimensions other than 3, and iii) beyond scalar quantities. This algorithm should find use in verifying analysis pipelines for higher-order statistics in upcoming galaxy redshift surveys such as DESI, Euclid, Roman, and Spherex, as well as intensity mapping. In particular it may be helpful in searches for parity violation in the 4PCF of these samples, for which producing initial conditions for N-body simulations is both costly and highly model-dependent at present, and so alternative methods such as that developed here are desirable

Cosmology and Nongalactic Astrophysics
Effective pair correlations of fractional powers of integers

Rafael Sayous

2023-06-06

Abstract:We study the statistics of pairs from the sequence $(n^\alpha)_{n\in\mathbb{N}^*}$, for every parameter $\alpha \in \, ]0,1[$. We prove the convergence of the empirical pair correlation measures towards a measure with an explicit density. In particular, when using the scaling factor $N\mapsto N^{1-\alpha}$, we prove that there exists an exotic pair correlation function which exhibits a level repulsion phenomenon. For other scaling factors, we prove that either the pair correlations are Poissonian or there is a total loss of mass. In addition, we give an error term for this convergence.

Number Theory,Functional Analysis
A class of binary sequences with two-valued cross correlations

Jianying Rong,Ting Li,Rui Hua,Xuemei Wang

DOI: https://doi.org/10.3934/math.2024442

2024-01-01

AIMS Mathematics

Abstract:Let $ p $ be an odd prime with $ p\equiv 7\pmod{12} $, $ \frac{p-1}2 $ be the least integer such that $ 2^{\frac{p-1}2}\equiv 1\pmod p $, and $ q = 2^{\frac{p-1}2} $. Let $ \alpha $ be a primitive element of the finite field $ \Bbb F_{q} $ and $ \beta = \alpha^{\frac{q-1}{p}} $. Suppose that $ \sigma = \sum_{i = 0}^2\beta^{m\zeta_3^i}\in \Bbb F_q^* $, where $ m\in \Bbb F_p^* $ and $ \zeta_3 $ is a $ 3 $rd root of unity in $ \Bbb F_p $. Let $ \{u_i\} = (\operatorname{Tr}_{q/2}(\sigma\beta^i))_{i = 0}^{q-2} $ be a binary sequence of period $ q-1 $. In this paper, we obtained the cross correlation distribution between two sequences $ \{u_i\} $ and its $ \frac{q-1}p $-decimation sequence, which is two-valued.

mathematics, applied
A p-adic Poissonian Pair Correlation Concept

Christian Weiss

DOI: https://doi.org/10.48550/arXiv.2308.15446

2023-08-30

Abstract:The pair correlation statistic is an important concept in real uniform distribution theory. Therefore, sequences in the unit interval with (weak) Poissonian pair correlations have attracted a lot of attention in recent time. The aim of this paper is to suggest a generalization to the p-adic integers and to prove some of its main properties. In particular, connections to the theory of p-adic discrepancy theory are discussed.

Number Theory
Fast analytical calculation of the random pair counts for realistic survey geometry

Michel-Andrès Breton,Sylvain de la Torre

DOI: https://doi.org/10.1051/0004-6361/202039603

2021-01-05

Abstract:Galaxy clustering is a standard cosmological probe that is commonly analysed through two-point statistics. In observations, the estimation of the two-point correlation function crucially relies on counting pairs in a random catalogue. The latter contains a large number of randomly distributed points, which accounts for the survey window function. Random pair counts can also be advantageously used for modelling the window function in the observed power spectrum. Since pair counting scales as $\mathcal{O}(N^2)$, where $N$ is the number of points, the computational time to measure random pair counts can be very expensive for large surveys. In this work, we present an alternative approach for estimating those counts that does not rely on the use of a random catalogue. We derived an analytical expression for the anisotropic random-random pair counts that accounts for the galaxy radial distance distribution, survey geometry, and possible galaxy weights. Considering the cases of the VIPERS and SDSS-BOSS redshift surveys, we find that the analytical calculation is in excellent agreement with the pair counts obtained from random catalogues. The main advantage of this approach is that the primary calculation only takes a few minutes on a single CPU and it does not depend on the number of random points. Furthermore, it allows for an accuracy on the monopole equivalent to what we would otherwise obtain when using a random catalogue with about 1500 times more points than in the data at hand. We also describe and test an approximate expression for data-random pair counts that is less accurate than for random-random counts, but still provides subpercent accuracy on the monopole. The presented formalism should be very useful in accounting for the window function in next-generation surveys, which will necessitate accurate two-point window function estimates over huge observed cosmological volumes.

Cosmology and Nongalactic Astrophysics
Sparsity meets correlation in Gaussian sequence model

Subhodh Kotekal,Chao Gao

2023-12-15

Abstract:We study estimation of an $s$-sparse signal in the $p$-dimensional Gaussian sequence model with equicorrelated observations and derive the minimax rate. A new phenomenon emerges from correlation, namely the rate scales with respect to $p-2s$ and exhibits a phase transition at $p-2s \asymp \sqrt{p}$. Correlation is shown to be a blessing provided it is sufficiently strong, and the critical correlation level exhibits a delicate dependence on the sparsity level. Due to correlation, the minimax rate is driven by two subproblems: estimation of a linear functional (the average of the signal) and estimation of the signal's $(p-1)$-dimensional projection onto the orthogonal subspace. The high-dimensional projection is estimated via sparse regression and the linear functional is cast as a robust location estimation problem. Existing robust estimators turn out to be suboptimal, and we show a kernel mode estimator with a widening bandwidth exploits the Gaussian character of the data to achieve the optimal estimation rate.

Statistics Theory,Methodology
The affinely invariant distance correlation

Johannes Dueck,Dominic Edelmann,Tilmann Gneiting,Donald Richards

DOI: https://doi.org/10.3150/13-bej558

IF: 1.5

2014-11-01

Bernoulli

Abstract:Székely, Rizzo and Bakirov (Ann. Statist. 35 (2007) 2769–2794) and Székely and Rizzo (Ann. Appl. Statist. 3 (2009) 1236–1265), in two seminal papers, introduced the powerful concept of distance correlation as a measure of dependence between sets of random variables. We study in this paper an affinely invariant version of the distance correlation and an empirical version of that distance correlation, and we establish the consistency of the empirical quantity. In the case of subvectors of a multivariate normally distributed random vector, we provide exact expressions for the affinely invariant distance correlation in both finite-dimensional and asymptotic settings, and in the finite-dimensional case we find that the affinely invariant distance correlation is a function of the canonical correlation coefficients. To illustrate our results, we consider time series of wind vectors at the Stateline wind energy center in Oregon and Washington, and we derive the empirical auto and cross distance correlation functions between wind vectors at distinct meteorological stations.

statistics & probability
Improved distance correlation estimation

Blanca E. Monroy-Castillo,Jácome,Ricardo Cao

2024-05-03

Abstract:Distance correlation is a novel class of multivariate dependence measure, taking positive values between 0 and 1, and applicable to random vectors of arbitrary dimensions, not necessarily equal. It offers several advantages over the well-known Pearson correlation coefficient, the most important is that distance correlation equals zero if and only if the random vectors are independent.

Computation,Statistics Theory
Elimination of Systematic Error in Digital Image Correlation Caused by Intensity Interpolation by Introducing Position Randomness to Subset Points

Yong Su,Qingchuan Zhang,Zheng Fang,Yaru Wang,Yang Liu,Shangquan Wu

DOI: https://doi.org/10.1016/j.optlaseng.2018.10.012

IF: 5.666

2018-01-01

Optics and Lasers in Engineering

Abstract:A strategy called random subset is presented to reduce systematic error in digital image correlation caused by intensity interpolation. In classical digital image correlation techniques, the subset points fall at integer locations; by contrast, the proposed random subset strategy imposes random offsets on the conventional locations, and consequently the subset points are located at non-integer positions. The introduction of position randomness to subset points makes the random subset strategy distinctly different from the classical digital image correlation technique. The random offsets can follow various probability distributions. In this work, two types of random offset were studied as examples, and the maximum of the random offsets was adjusted to modify performance. Using the random subset strategy, random-subset-based subpixel registration algorithms were developed in this study, including the random-subset-based forward additive Gaussian Newton algorithm and the random-subset-based inverse compositional Gauss Newton algorithm. The measurement accuracy of the random subset-based subpixel registration algorithms was investigated. Numerical simulations and real-world experiments demonstrated that as the maximum of the random offsets increases, the amplitude of the sinusoidal-shaped systematic error changes from positive to negative. Hence, if an appropriate random offset is selected, systematic error can be considerably reduced. In fact, the accuracy enhancement is greater than one order of magnitude, and systematic error is virtually eliminated.
Decay of Multi-point Correlation Functions in

Rui Han,Fan Yang

DOI: https://doi.org/10.1007/s00220-023-04884-6

IF: 2.361

2024-02-14

Communications in Mathematical Physics

Abstract:We prove multi-point correlation bounds in for arbitrary with symmetrized distances, answering open questions proposed by Sims–Warzel (Commun Math Phys 347(3):903–931, 2016) and Aza–Bru–Siqueira Pedra (Commun Math Phys 360(2):715–726, 2018). As applications, we prove multi-point correlation bounds for the Ising model on , and multi-point dynamical localization in expectation for uniformly localized disordered systems, which provides the first examples of this conjectured phenomenon by Bravyi–König (Commun Math Phys 316(3):641–692, 2012) .

physics, mathematical
Measuring Linear Correlation Between Random Vectors

Giovanni Puccetti

DOI: https://doi.org/10.2139/ssrn.3116066

2018-01-01

SSRN Electronic Journal

Abstract:We introduce a new scalar coefficient to measure linear correlation between random vectors which preserves all the relevant properties of Pearson’s correlation in arbitrary dimensions. The new measure and its bounds are derived from a mass transportation approach in which the expected inner product of two random vectors is taken as a measure of their covariance and then standardized by the maximal attainable value given their marginal covariance matrices. In several simulative studies we show the limiting distribution of the empirical estimator of the newly defined index and of the corresponding rank correlation. A comparative study shows that our proposed correlation, though derived from a novel approach, behaves similarly to some of the multivariate dependence notions recently introduced in the literature. Throughout the paper, we also give some auxiliary results of independent interest in matrix analysis and mass transportation theory, including an improvement to the Cauchy-Schwarz inequality for positive definite covariance matrices.

Increased accuracy of the two-point correlation function at no extra cost

Pair Counting without Binning -- A New Approach to Correlation Functions in Clustering Statistics

The 2-point correlation function covariance with fewer mocks

corr2D - Implementation of Two-Dimensional Correlation Analysis in R

Improved two-point correlation function estimates using glass-like distributions as a reference sample

A Set of Efficient Methods to Generate High-Dimensional Binary Data with Specified Correlation Structures

A new method for the robust characterisation of pairwise statistical dependency between point processes

Calculating pair-correlations from random particle configurations

An Improved Estimator for the Correlation Function of 2D Nonlinear Sigma Models

On the smallest gap in a sequence with Poisson pair correlations

Algorithm to Produce a Density Field with Given Two, Three, and Four-Point Correlation Functions

Effective pair correlations of fractional powers of integers

A class of binary sequences with two-valued cross correlations

A p-adic Poissonian Pair Correlation Concept

Fast analytical calculation of the random pair counts for realistic survey geometry

Sparsity meets correlation in Gaussian sequence model

The affinely invariant distance correlation

Improved distance correlation estimation

Elimination of Systematic Error in Digital Image Correlation Caused by Intensity Interpolation by Introducing Position Randomness to Subset Points

Decay of Multi-point Correlation Functions in

Measuring Linear Correlation Between Random Vectors