Short Note on Ridge Regression

Introduction

I am doing a couple of assignments involving penalized estimators such as Ridge regression, and I wanted to do a short derivation of its asymptotic covariance. In comparison to the existing resources which I could find, some details are left out, which I wanted to recapitulate more clearly. I’ll also contrast the variance of the Ridge estimator to the variance of the OLS estimator, illustrating a fact that also comes to the surface in many other resources, namely that the variance of the Ridge estimator is smaller than that of the OLS estimator.

Setting

I assume non-stochastic regressors \(X\), and a model with \(Y = X\beta + \epsilon\), \(\epsilon \sim \mathcal{N}(0, \sigma^2 I)\) with \( \mathbb{E}[X^T \epsilon] = 0 \).

The ridge estimator can be expressed as:

$$ \hat{\beta}_{R} = (X^T X + \lambda I)^{-1} X^T y $$

It is easy to show that the Ridge estimator is biased for \(\lambda \neq 0\) by evaluating the expected value.

Consistency of the Ridge Estimator

Doing so also allows us to express \( \hat{\beta}_{R} \) as:

$$ \hat{\beta}_R = (X^T X + \lambda I)^{-1} X^T X \beta $$

Taking the \( \text{plim} \) of this expression and applying Slutsky’s theorem then gives:

$$ \text{plim} \left((X^T X + \lambda I)^{-1} X^T X \beta \right) = \text{plim} (\frac{1}{n} X^T X + \frac{1}{n} \lambda I)^{-1} \cdot \text{plim} (\frac{1}{n} X^T X) \beta $$

After realizing that

$$ \text{plim} ( \frac{1}{n} \lambda I) = 0 $$

the above expression simplifies to \(\beta\), thus showing consistency.

Asymptotic Variance of the Ridge Estimator

The asymptotic variance variance of the Ridge regression around its \(\text{plim}\) can be obtained by rewriting the estimator in the following form:

$$ \hat{\beta}_R = (X^T X + \lambda I)^{-1} X^T y $$

$$ = (X^TX + \lambda I)^{-1} X^T (X\beta + \epsilon) $$

$$ = (X^TX + \lambda I)^{-1} X^T \beta + (X^T X + \lambda I)^{-1} X^T \epsilon $$

Which by the CLT converges to its \(\text{plim}\). The variance is then determined by its second part, since the first part is stochastic.

• First, then, according to the (a) CLT, \( X^T \epsilon \) converges to its \(\text{plim}\) , which is zero by assumption, with a variance being equal to \(\sigma^2 X^T X\). Then, by the product limit normal rule (Cameron & Trivedi, 2005, Theorem A.17), the variance of \(\hat{\beta}_R\) is then equal to:

$$ \text{Var} (\hat{\beta}_R) = \sigma^2 (X^T X + \lambda I)^{-1} X^T X (X^T X + \lambda I)^{-1} $$

which can also be expressed as:

$$ \text{Var} (\hat{\beta}_R) = \sigma^2 (X^T X + \lambda I)^{-1} X^T X (X^T X)^{-1} X^T X (X^T X + \lambda I)^{-1} $$

Comparison of Variance with OLS Estimator

Now, I show the positive semidefiniteness of the matrix Var \( \beta_{OLS} \) - Var \( \beta_{R} \):

• Taking the previous expression, and defining \( W = X^T X (X^T X + \lambda I )^{-1}\), we can rewrite Var \( \hat{\beta}_{R}\) in a simple form:

$$ \text{Var} ( \hat{\beta}_{R} ) = \sigma^2 W^T (X^T X)^{-1} W $$

The difference between Var \( \beta_{OLS} \) - Var \( \beta_{R} \) is then:

$$ \text{Var} (\hat{\beta_{OLS}}) - \text{Var} (\hat{\beta}_{R}) = \sigma^2 (X^T X)^{-1} - \sigma^2 W^T (X^T X)^{-1} W \newline = \sigma^2 \left( (X^T X)^{-1} - W^T (X^T X)^{-1} W \right) $$

It remains to show that

$$ \left( (X^T X)^{-1} - W^T (X^T X)^{-1} W \right) $$

is a positive semi-definite matrix. First, since \( X^T X \) is p.s.d., \( (X^T X)^{-1}\) is also p.s.d. (A short proof is comparing the eigenvalues of a matrix and its inverse). Also, if you add \( \lambda I \) to a matrix, its eigenvalues increase with \( \lambda \). Hence with \( \lambda > 0 \), we increase the already positive eigenvalues, and \( W \) is also p.s.d.

After some derivation, we can show that the difference in variances is equal to the following quadratic form:

$$ \sigma^2 (X^T X + \lambda I)^{-1} \left[ 2 \lambda I + \lambda^2 (X^T X)^{-1} \right] (X^T X + \lambda I)^{-1} $$

Since, by the preceding discussion, all matrices here are p.s.d., the final variance is positive semi-definite and the variance of the OLS estimator is larger than the variance of the Ridge estimator.

Conclusion

In this post, I have set out some properties of the Ridge estimator, arguably the easiest to understand shrinkage estimator. I have focused on some standard theoretical results, and try to explain this in a way that works for me. Thank you for reading!