Nilpotent elements of group algebra $\Bbb CG$

You can get a lot more insight by actually looking at what the 2-dimensional representation is explicitly. It's just the usual action of $D_8$ as symmetries of a square, so $a$ maps to the matrix $A=\begin{pmatrix}0 & -1 \\ 1 & 0 \end{pmatrix}$ and $b$ maps to $B=\begin{pmatrix}-1 & 0 \\ 0 & 1\end{pmatrix}$. We can now just fiddle about to find a combination of these matrices that gives $\begin{pmatrix} 0 & 1 \\ 0 & 0\end{pmatrix}$. Specifically, we have $$BA=\begin{pmatrix}0 & 1 \\ 1 & 0\end{pmatrix}$$ so $$\frac{BA-A}{2}=\begin{pmatrix} 0 & 1 \\ 0 & 0\end{pmatrix}.$$

Now this doesn't tell us that $\frac{ba-a}{2}$ is nilpotent, only that its image in the 2-dimensional representation is nilpotent. We can eliminate its image in the other representations by multiplying by the idempotent $\frac{1-a^2}{2}$ which you found. So, we conclude that $$\frac{ba-a}{2}\cdot\frac{1-a^2}{2}$$ is a nontrivial nilpotent element of $\mathbb{C}D_8$.

Just to verify, we can square it: $$\begin{align*} \left(\frac{ba-a}{2}\cdot\frac{1-a^2}{2}\right)^2 &= \left(\frac{ba-a}{2}\right)^2\cdot\left(\frac{1-a^2}{2}\right)^2 \\ &=\frac{baba-ba^2-aba+a^2}{4}\cdot \frac{1-a^2}{2} \\ &=\frac{1-ba^2-b+a^2}{4}\cdot \frac{1-a^2}{2} \\ &=\frac{1-a^2-ba^2+ba^4-b+ba^2+a^2-a^4}{8} \\ &=\frac{1-a^2-ba^2+b-b+ba^2+a^2-1}{8} \\ &= 0. \end{align*} $$

Just to provide more context.

Let $G$ be a finite group, and let $H\le G$ be a non-trivial subgroup, which is not normal. Then if $$ \alpha_H=\frac{1}{|H|}\sum h$$ you can check that $\alpha_H^2=\alpha_H$, and thus $(1-\alpha_H)\alpha_H=0$. Let $g\in G$ be such that $g\notin N_G(H)$. Then we have $$ \alpha_Hg\neq\alpha_Hg\alpha_H $$ and so if we define $\beta_H=\alpha_Hg(1-\alpha_H)$, then $\beta_H\neq0$ but $\beta_H^2=0$.

This only works if we can find a non-normal subgroup $H$. The finite groups that have every subgroup normal are well-known, and the non-abelian ones look like $A\times Q_8$, where $A$ is abelian and $Q_8$ is the quaternion group. So to finish this argument for all non-abelian groups, it suffices to exhibit a nilpotent element in $\mathbb{C}Q_8$.

I'll leave this to you, only noting that you cannot do it over $\mathbb{Q}$, since $\mathbb{Q}Q_8$ has no nontrivial nilpotents. But it can be done over $\mathbb{Q}(i)$.