Adding Absolute value to a complex number: $ z+| z|=2+8i$
I would go about this differently. Since $|z| \in \mathbb{R}$, you know that $b=8$ immediately since $bi$ is the only imaginary term on the left and $8i$ - on the right.
Now the only thing is to find $a$...
UPDATE
We have the equation $$a + \sqrt{a^2+64} = 2$$ (hence $a<0$), which implies $$\sqrt{a^2+64} = 2-a$$ and now squaring will yield the desired result.
Yet another way: $\,z=2+8i-|z|\,$, so $\,\bar z =2-8i-|z|\,$, then multiplying the two:
$$\require{cancel} z \bar z = (2+8i-|z|)(2-8i-|z|) \;\iff\; \cancel{|z|^2} = \cancel{|z|^2} - 4|z| + 68 \;\iff\; |z| = 17 $$
Then, substituting back in the first equation: $\,z=2+8i-|z|=\ldots\,$
$a+bi+\sqrt{a^2+b^2} = 2 + 8i$ so
$a + \sqrt{a^2 + b^2} = 2$ and $b = 8$.
So $a + \sqrt{a^2 + 64} = 2$
So $\sqrt{a^2 + 64} = 2- a$
$a^2 + 64 = 4 -4a + a^2$
$4a = -60$
$a = -15$.
$z = -15 + 8i$.
....
To do what you were attempting
You have to realize that the $Re(z) = a + \sqrt{a^2 + b^2}$ and $Im(z) = b$. I think somehow you were thinking there were threee parts $Re(z)=a$ and $Im(z) = b$ and some $Weird(z) = \sqrt{a^2 + b^2}$ and that $z\overline z = Re^2(z) - Im^2(z) + Weird^2(z)$. That simply isn't true....
$(a+bi+\sqrt{a^2+b^2})(a - bi +\sqrt{a^2 + b^2}) = (2 + 8i)(2-8i)$
$(a + \sqrt{a^2 + b^2})^2 - b^2 = 4 - 64$
$2a^2 + b^2 + 2a\sqrt{a^2 + b^2} -b^2 = -60$
$a^2 + a \sqrt{a^2 + b^2} = -30$
which is a pain to solve but can be done.