Projection Matrix

Projection matrix of \(X\) maps a vector to the column space spanned by the matrix \(X\).

If the projection of \(y\) on column space of \(X\) is \(\hat{y} = Xa\), then the residual \(y - Xa\) must be orthogonal to the column space of \(X\) i.e.

\begin{align*} X^T (y - X a) = 0 \\ X^T y - X^T X a = 0 \\ a = (X^T X)^{-1} X^T y \\ \end{align*}

So,

\begin{align*} \hat{y} = X a \\ \hat{y} = X (X^T X)^{-1} X^T y \end{align*}

Thus the projection matrix \(P = X (X^T X)^{-1} X^T\) such that \(\hat{y} = P y\)

Properties:

1. It is symmetric \(P^T = P\)
2. It is idempotent \(P^2 = P\)
3. If \(X\) is of rank \(r\) then \(P\) is also of rank \(r\) and its eignevalues are \(r\) ones and rest zeroes.
4. \(X\) is invariant under \(P\), i.e. \(PX = X\)