Independent Processes in Signal Processing

This jupyter notebook is part of a collection of notebooks on various topics of Digital Signal Processing. Please direct questions and suggestions to Sascha.Spors@uni-rostock.de.

The independence of random signals is a desired property in many applications of statistical signal processing, as well as uncorrelatedness and orthogonality. The concept of independence is introduced in the following together with a discussion of the links to uncorrelatedness and orthogonality.

Definition

Two stochastic events are said to be independent if the probability of occurrence of one event is not affected by the occurrence of the other event. Or more specifically, if their joint probability equals the product of their individual probabilities. In terms of the bivariate probability density function (PDF) of two continuous-amplitude real-valued random processes $x[k]$ and $y[k]$ this reads

$$\begin{array}{c}{p}_{xy}({\theta }_x,{\theta }_y,k_x,k_y)=p_x({\theta }_x,k_x)\cdot p_y({\theta }_y,k_y)\end{array}$$

where $p_x({\theta }_x,k_x)$ and $p_y({\theta }_y,k_y)$ denote the univariate (marginal) PDFs of the random processes for the time-instances $k_x$ and $k_y$, respectively. The bivariate PDF of two independent random processes is given by the multiplication of their univariate PDFs. It follows that the second-order ensemble average for a linear mapping is given as

$$\begin{array}{c}E\{x[k_x]\cdot y[k_y]\}=E\{x[k_x]\}\cdot E\{y[k_y]\}\end{array}$$

The linear second-order ensemble average of two independent random signals is equal to the multiplication of their linear first-order ensemble averages. For jointly wide-sense stationary (WSS) processes, the bivariate PDF does only depend on the difference $\kappa =k_x-k_y$ of the time instants. Hence, two jointly WSS random signals are independent if

$$\begin{array}{c}\begin{array}{rl}{p}_{xy}({\theta }_x,{\theta }_y,\kappa )& =p_x({\theta }_x,k_x)\cdot p_y({\theta }_y,k_x-\kappa )\\ & =p_x({\theta }_x)\cdot p_y({\theta }_y,\kappa )\end{array}\end{array}$$

Above bivariate PDF is rewritten using the definition of conditional probabilities in order to specialize the definition of independence to one WSS random signal $x[k]$

$$\begin{array}{c}{p}_{xy}({\theta }_x,{\theta }_y,\kappa )=p_{y|x}({\theta }_x,{\theta }_y,\kappa )\cdot p_x({\theta }_x)\end{array}$$

where $p_{y|x}({\theta }_x,{\theta }_y,\kappa )$ denotes the conditional probability that $y[k-\kappa ]$ takes the amplitude value ${\theta }_y$ under the condition that $x[k]$ takes the amplitude value ${\theta }_x$. Under the assumption that $y[k-\kappa ]=x[k-\kappa ]$ and substituting ${\theta }_x$ and ${\theta }_y$ by ${\theta }_1$ and ${\theta }_2$, independence for one random signal is defined as

$$\begin{array}{c}{p}_{xx}({\theta }_1,{\theta }_2,\kappa )=\{\begin{array}{ll}{p}_x({\theta }_1)\cdot \delta ({\theta }_2-{\theta }_1)& \text{for }\kappa =0\\ p_x({\theta }_1)\cdot p_x({\theta }_2,\kappa )& \text{for }\kappa \ne 0\end{array}\end{array}$$

since the conditional probability $p_{x[k]|x[k-\kappa ]}({\theta }_1,{\theta }_2,\kappa )=\delta ({\theta }_2-{\theta }_1)$ for $\kappa =0$ since this represents a sure event. The bivariate PDF of an independent random signal is equal to the product of the univariate PDFs of the signal and the time-shifted signal for $\kappa \ne 0$. A random signal for which this condition does not hold shows statistical dependencies between samples. These dependencies can be exploited for instance for coding or prediction.

Example - Comparison of bivariate PDF and product of marginal PDFs

The following example estimates the bivariate PDF $p_{xx}({\theta }_1,{\theta }_2,\kappa )$ of a WSS random signal $x[k]$ by computing its two-dimensional histogram. The univariate PDFs $p_x({\theta }_1)$ and $p_x({\theta }_2,\kappa )$ are additionally estimated. Both the estimated bivariate PDF and the product of the two univariate PDFs $p_x({\theta }_1)\cdot p_x({\theta }_2,\kappa )$ are plotted for different $\kappa$.

Exercise

• With the given results, how can you evaluate the independence of the random signal?
• Can the random signal assumed to be independent?

Solution: According to the definition of independence, the bivariate PDF and the product of the univariate PDFs has to be equal for $\kappa \ne 0$. This is obviously not the case for $\kappa =2$. Hence, the random signal is not independent in a strict sense. However for $\kappa =20$ the condition for independence is sufficiently fulfilled, considering the statistical uncertainty due to a finite number of samples.

Independence versus Uncorrelatedness

Two continuous-amplitude real-valued jointly WSS random processes $x[k]$ and $y[k]$ are termed as uncorrelated if their cross-correlation function (CCF) is equal to the product of their linear means, ${\phi }_{xy}[\kappa ]={\mu }_x\cdot {\mu }_y$. If two random signals are independent then they are also uncorrelated. This can be proven by introducing above findings for the linear second-order ensemble average of independent random signals into the definition of the CCF

$$\begin{array}{c}{\phi }_{xy}[\kappa ]=E\{x[k]\cdot y[k-\kappa ]\}=E\{x[k]\}\cdot E\{y[k-\kappa ]\}={\mu }_x\cdot {\mu }_y\end{array}$$

where the last equality is a consequence of the assumed wide-sense stationarity. The reverse, that two uncorrelated signals are also independent does not hold in general from this result.

The auto-correlation function (ACF) of an uncorrelated signal is given as ${\phi }_{xx}[\kappa ]={\mu }_x^2+{\sigma }_x^2\cdot \delta [\kappa ]$. Introducing the definition of independence into the definition of the ACF yields

$$\begin{array}{c}\begin{array}{rl}{\phi }_{xx}[\kappa ]& =E\{x[k]\cdot x[k-\kappa ]\}\\ & =\{\begin{array}{ll}E\{x^2[k]\}& \text{for }\kappa =0\\ E\{x[k]\}\cdot E\{x[k-\kappa ]\}& \text{for }\kappa \ne 0\end{array}\\ & =\{\begin{array}{ll}{\mu }_x^2+{\sigma }_x^2& \text{for }\kappa =0\\ {\mu }_x^2& \text{for }\kappa \ne 0\end{array}\\ & ={\mu }_x^2+{\sigma }_x^2\delta [\kappa ]\end{array}\end{array}$$

where the result for $\kappa =0$ follows from the bivariate PDF $p_{xx}({\theta }_1,{\theta }_2,\kappa )$ of an independent signal, as derived above. It can be concluded from this result that an independent random signal is also uncorrelated. The reverse, that an uncorrelated signal is independent does not hold in general.

Independence versus Orthogonality

In geometry, two vectors are said to be orthogonal if their dot product equals zero. This definition is frequently applied to finite-length random signals by interpreting them as vectors. The relation between independence, correlatedness and orthogonality is derived in the following.

Let's assume two continuous-amplitude real-valued jointly wide-sense ergodic random signals $x_N[k]$ and $y_M[k]$ with finite lengths $N$ and $M$, respectively. The CCF ${\phi }_{xy}[\kappa ]$ between both can be reformulated as follows

$$\begin{array}{c}\begin{array}{rl}{\phi }_{xy}[\kappa ]& =\frac{1}{N}\sum _{k=0}^{N-1}{x}_N[k]\cdot y_M[k-\kappa ]\\ & =\frac{1}{N}<{\mathbf{x}}_N,{\mathbf{y}}_M[\kappa ]>\end{array}\end{array}$$

where $<\cdot ,\cdot >$ denotes the dot product. The $(N+2M-2)\times 1$ vector ${\mathbf{x}}_N$ is defined as

$${\mathbf{x}}_N={\left[0_{(M-1)\times 1}^T,x[0],x[1],\dots ,x[N-1],0_{(M-1)\times 1}^T\right]}^T$$

where $0_{(M-1)\times 1}$ denotes the zero vector of length $M-1$. The $(N+2M-2)\times 1$ vector ${\mathbf{y}}_M[\kappa ]$ is defined as

$${\mathbf{y}}_M={\left[0_{\kappa \times 1}^T,y[0],y[1],\dots ,y[M-1],0_{(N+M-2-\kappa )\times 1}^T\right]}^T$$

It follows from above definition of orthogonality that two finite-length random signals are orthogonal if their CCF is zero. This implies that at least one of the two signals has to be mean free. It can be concluded further that two independent random signals are also orthogonal and uncorrelated if at least one of them is mean free. The reverse, that orthogonal signals are independent, does not hold in general.

The concept of orthogonality can also be extended to one random signal by setting ${\mathbf{y}}_M[\kappa ]={\mathbf{x}}_N[\kappa ]$. Since a random signal cannot be orthogonal to itself for $\kappa =0$, the definition of orthogonality has to be extended for this case. According to the ACF of a mean-free uncorrelated random signal $x[k]$, self-orthogonality may be defined as

$$\begin{array}{c}\frac{1}{N}<{\mathbf{x}}_N,{\mathbf{x}}_N[\kappa ]>=\{\begin{array}{ll}{\sigma }_x^2& \text{for }\kappa =0\\ 0& \text{for }\kappa \ne 0\end{array}\end{array}$$

An independent random signal is also orthogonal if it is zero-mean. The reverse, that an orthogonal signal is independent does not hold in general.

Example - Computation of cross-correlation by dot product

This example illustrates the computation of the CCF by the dot product. First, a function is defined which computes the CCF by means of the dot product

Now the CCF is computed using different methods: computation by the dot product and by the built-in correlation function. The CCF is plotted for the computation by the dot product, as well as the difference (magnitude) between both methods. The resulting difference is in the typical expected range due to numerical inaccuracies.

Copyright

This notebook is provided as Open Educational Resource. Feel free to use the notebook for your own purposes. The text is licensed under Creative Commons Attribution 4.0, the code of the IPython examples under the MIT license. Please attribute the work as follows: Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples.