How Does $ {L}_{1} $ Regularization Present Itself in Gradient Descent?

It changes the direction you descent towards.

You may have a look at this PDF - Steepest Descent Direction for Various Norms.
It shows the direction for few different norms.


The problem is that the gradient of the norm does not exist at $0$, so you need to be careful

$$ E_{L_1} = E + \lambda\sum_{k=1}^N|\beta_k| $$

where $E$ is the cost function (E stands for error), which I will assume you already know how to calculate the gradient for.

As for the regularization term, note that if $\beta_k > 0$ then $|\beta_k| = \beta_k$ and the gradient is $+1$, similarly when $\beta_k < 0$ the gradient is $-1$, so in summary

$$ \frac{\partial |\beta_k|}{\partial \beta_l} = {\rm sgn}(\beta_k)\delta_{kl} $$

so that

$$ \frac{\partial E_{L_1}}{\partial \beta_l} = \frac{\partial E}{\partial \beta_l} + \lambda\sum_{k=1}^N {\rm sgn}(\beta_k)\delta_{kl} = \frac{\partial E}{\partial \beta_l} + \lambda {\rm sgn}(\beta_l) $$

Tags:

Optimization

Gradient Descent

Algorithms

Regularization

Vector Analysis

Related

Moving from $x^2+x+1=0$ to $x+1+\frac1x=0$ seems to introduce a solution, how is this possible? $p(x)x^n+q(x)(1-x)^n=1$ for some $p(x), q(x)\in \mathbb{Z}[x]$, what explicitly $p(x), q(x)$ are? In a Zener card test using a 25 -card pack and no replacement, what is the expected score if we try to minimise correct guesses? Structures with $x*(y*z) = y*(x*z)$ Show there is no $n>0$ where $n\in \mathbb{Z}$ such that $n$, $n+10$, $n+20$, $n+30$ are all prime Give me an example of a topological manifold which is not a smooth manifold. Product rule for gradient Are there examples of conjectures that turned out to be false despite of strong heuristics? Can we prove $\sin(x)^2 + \cos(x)^2 = 1$ using just their series? What are all finite groups such that all isomorphic subgroups are identical? $f\circ g(x)$ and $g\circ f(x)$ are given; find $f$ and $g$ Example of an infinite dimensional Hilbert space that is not an RKHS

Recent Posts