How do I exactly project a vector onto a subspace?

I will talk about orthogonal projection here.

When one projects a vector, say $v$, onto a subspace, you find the vector in the subspace which is "closest" to $v$. The simplest case is of course if $v$ is already in the subspace, then the projection of $v$ onto the subspace is $v$ itself.

Now, the simplest kind of subspace is a one dimensional subspace, say the subspace is $U = \operatorname{span}(u)$. Given an arbitrary vector $v$ not in $U$, we can project it onto $U$ by $$v_{\| U} = \frac{\langle v , u \rangle}{\langle u , u \rangle} u$$ which will be a vector in $U$. There will be more vectors than $v$ that have the same projection onto $U$.

Now, let's assume $U = \operatorname{span}(u_1, u_2, \dots, u_k)$ and, since you said so in your question, assume that the $u_i$ are orthogonal. For a vector $v$, you can project $v$ onto $U$ by $$v_{\| U} = \sum_{i =1}^k \frac{\langle v, u_i\rangle}{\langle u_i, u_i \rangle} u_i = \frac{\langle v , u_1 \rangle}{\langle u_1 , u_1 \rangle} u_1 + \dots + \frac{\langle v , u_k \rangle}{\langle u_k , u_k \rangle} u_k.$$


Take a basis $\{v_1, \dots, v_n\}$ for the "signal subspace" $V$. Let's assume $V$ is finite dimensional for simplicity and practical purposes, but you can generalize to infinite dimensions. Let's also assume the basis is orthonormal.

The projection of your signal $f$ onto the subspace $V$ is just

$$\mathrm{proj}_V(f) = \sum_{i=1}^n \langle f, v_i \rangle v_i$$

and $f = \mathrm{proj}_V(f) + R(f)$, where $R(f)$ is the remainder, or orthogonal complement, which will be 0 if $f$ lies in the subspace $V$.

The $i$-th term of the sum, $\langle f, v_i\rangle$, is the projection of $f$ onto the subspace spanned by the $i$-th basis vector. (Note, if the $v_i$ are orthogonal, but not necessarily orthonormal, you must divide the $i$-th term by $\|v_i\|^2$.)


The projection of the vector is the vector belonging to the subspace that best approximates the former, i.e. such that the (squared) norm of the difference is the smallest.

WLOG take a subspace spanned by three vectors. We minimize for the coordinates $u,v,w$

$$\epsilon=(\vec d-u\vec a-v\vec b-w\vec c)^2.$$

This is achieved by canceling the gradient (factor $2$ omitted),

$$\begin{cases} \dfrac{d\epsilon}{du}=\vec a\cdot(\vec d-u\vec a-v\vec b-w\vec c)=0,\\ \dfrac{d\epsilon}{dv}=\vec b\cdot(\vec d-u\vec a-v\vec b-w\vec c)=0,\\ \dfrac{d\epsilon}{dw}=\vec c\cdot(\vec d-u\vec a-v\vec b-w\vec c)=0.\\ \end{cases}$$ giving a linear system of three equations in three unknonws.

In the special case that $\vec a,\vec b,\vec c$ form an orthonormal basis, we simply have

$$u=\vec d\cdot\vec a,v=\vec d\cdot\vec b,w=\vec d\cdot\vec c,$$ and the projection is

$$\vec u^*=(\vec d\cdot\vec a)\vec a+(\vec d\cdot\vec b)\vec b+(\vec d\cdot\vec c)\vec c.$$