Fitting a curve: degree, overfitting and ridge

Least squares with polynomial features, θ = (ΦᵀΦ + λI)⁻¹Φᵀy. Change the degree and the regularization.

Polynomial regression with ridge

degree M ridge λ training points

Try thisWith 12 points, go to degree 11: the curve passes through every point and test error explodes. Now raise λ to about 0.01: the same degree-11 model becomes reasonable. That is MAP with a Gaussian prior.