Add a penalty to linear regression to keep the weights small. The step people get wrong is not the penalty term itself — it is leaving the intercept out of it.

Why the intercept is not penalized

The intercept shifts every prediction up or down as a block, so it says nothing about model complexity. Penalize it and the model becomes systematically biased the further the target mean sits from zero. That is why standard implementations always exclude w[0] from the penalty.

Shared setup

Same as [[algo/mle-interview/linear-regression-scratch|linear regression]] — prepend a column of ones to the design matrix, start from the zero vector, and apply w = w - lr * grad exactly epochs times.

The base gradient is unchanged:

$g_{\text{mse}} = \frac{2}{n}A^\top(Aw - y)$

Ridge (L2)

$g = g_{\text{mse}} + 2\alpha w', \qquad w' = (0, w_1, \dots, w_d)$

w' is the weight vector with the intercept slot zeroed out.

Lasso (L1)

The absolute value is not differentiable at zero, so use a subgradient.

$g = g_{\text{mse}} + \alpha\,\text{sign}(w'), \qquad \text{sign}(0) = 0$

Lasso's coefficient carries no factor of 2 — that is the difference from ridge.

Return

• fit_ridge(X, y, lr, epochs, alpha) / fit_lasso(X, y, lr, epochs, alpha)
• A list of length d+1, every float rounded to 6 decimal places.
• With alpha=0 both must reproduce plain linear regression exactly.

Implement fit_ridge(X, y, lr, epochs, alpha).

• Start from the zero vector, update exactly epochs times.
• grad = (2/n) A.T @ (A @ w - y) + 2 * alpha * w'
• w' is the weight vector with the intercept slot zeroed — no penalty lands on w[0].
• Round each element with round(v, 6).

Get alpha=0 matching plain linear regression first; then you can verify the penalty term on its own.