Interview Problems: Correlation

Problems:

(1) Why correlation matrix is positive semi-definite? 

(2) For n random variables, if all their pairwise correlations are $\rho$, what is the range of $\rho$?

(3) For three random variables X, Y and Z, the correlation between X and Y is $a$ and the correlation between X and Z is $b$. What is the range of correlation between Y and Z? Solve it in three ways.

(4) For two covariance matrices A and B, is AB a covariance matrix when AB=BA. 

Solution:

(1) Since $\text{Var}(\sum_{i} a_iX_i)=\sum_{i,j}a_ia_j\text{Cov}(X_i, X_j) \geq 0$, covariance matrix is positive semi-definite. So does correlation matrix since it is a covariance matrix when all random variables have unit variance.  

(2) The correlation matrix is of the form $\boldsymbol{\Sigma}=(1-\rho)\mathbf{I}+\rho\, \mathbf{e}\mathbf{e}^T$ where $\mathbf{e}\equiv[1, \cdots, 1]^T$. $\mathbf{e}\mathbf{e}^T$ has one eigenvector $\mathbf{e}$ with eigenvalue $n$ and $n-1$ eigenvectors that are normal to $\mathbf{e}$ with eigenvalues 0. As a result, $\boldsymbol{\Sigma}$ has one eigenvalue $1+(n-1)\rho$ and $n-1$ eigenvalues $1-\rho$, which should cannot be negative since $\boldsymbol{\Sigma}$ is semi-positive definite. That is $-\frac{1}{n-1}\leq \rho \leq 1$.

(3) Method 1: Suppose the correlation between $Y$ and $Z$ is $c$. The correlation matrix $\left[\begin{array}{ccc} 1 & a & b \\ b & 1 & c\\ a & c & 1 \end{array}\right]\succeq 0$ gives the constraint $c^2-2abc+a^2+b^2-1\leq 0$. As a result, $ab-\sqrt{(1-a^2)(1-b^2)}\leq c \leq ab +\sqrt{(1-a^2)(1-b^2)}$.

Method 2: Correlation is cosine when the inner product is taken as covariance. So the bounds of correlation are $\cos\left(\arccos a + \arccos b\right)$ and $\cos\left(\arccos a - \arccos b\right)$, that is, $ab\pm \sqrt{(1-a^2)(1-b^2)}$.

Method 3: Without loss of generality, consider the case of all unit variance. Let $Y\equiv aX+\sqrt{1-a^2} U$ and $Z\equiv bX+\sqrt{1-b^2} V$ with $\text{Cov}(X, U)=\text{Cov}(X,V)=0$. As a result, $\text{Cov}(Y, Z)=ab+\sqrt{(1-a^2)(1-b^2)}\,\text{Cov}(U, V)$ and $|\text{Cov(U, V)}|\leq 1$.

(4) Yes. AB is symmetric and its eigenvalues are positive semi-definite. Note that A and B commute, and thus can be diagonalized simultaneously. As a result, a eigenvalue of AB are the product of the eigenvalues of A and B with the same eigenstate.