Evaluate $\tan^{2}(20^{\circ}) + \tan^{2}(40^{\circ}) + \tan^{2}(80^{\circ})$

Method $1:$

We know $$ \tan^2A=\frac{1-\cos2A}{1+\cos2A} $$

Let us find the cubic equation whose roots are $\cos40^\circ, \cos80^\circ, \cos160^\circ$.

As $\cos(3\cdot 40^{\circ})=\cos120^{\circ}=-\frac{1}{2}$ or, $4\cos^340^{\circ} -3\cos40^{\circ}=-\frac{1}{2}$.

So, $\cos40^{\circ} $ is a root of $$ 4x^3-3x=-\frac12\implies 8x^3-6x+1=0 $$ Similarly, $\cos80^{\circ},\cos160^{\circ}$ are also the roots of $ 8x^3-6x+1=0 $ (Another derivation can be found at the bottom)

If we replace $x$ with $\dfrac{1-y}{1+y}$, the sum of the roots of the new equation in $y$ will give us the desired value.

Method $2:$ (Inspired by Zarrax's answer)

Observe that $\tan(3\cdot20^\circ)=\tan60^\circ=\sqrt3$

$\tan(3\cdot40^\circ)=\tan120^\circ=\tan(180^\circ-60^\circ)=-\tan60^\circ=-\sqrt3$ $\iff \tan\{3(-40^\circ)\}=\sqrt3$

and $\tan(3\cdot80^\circ)=\tan240^\circ=\tan(180^\circ+60^\circ)=\tan60^\circ=\sqrt3$

$$\text{As }\tan3\theta=\frac{3\tan\theta-\tan^3\theta}{1-3\tan^2\theta}$$

$$\text{the roots of the equation } t^3-3\sqrt3t^2-3t+\sqrt3=0 (\text{ Putting } \tan3\theta=\sqrt3)$$ will be $\tan20^\circ,\tan(-40^\circ)=-\tan40^\circ, \tan80^\circ$

Using Vieta's formulas, $$\tan20^\circ+(-\tan40^\circ)+\tan80^\circ=\frac{3\sqrt3}1$$

$$\text{and } \tan20^\circ(-\tan40^\circ)+\tan20^\circ\cdot\tan80^\circ+\tan80^\circ(-\tan40^\circ)=-3$$

$$\text{So,}\tan^220^\circ+\tan^240^\circ+\tan^280^\circ =(\tan20^\circ)^2+(-\tan40^\circ)^2+(\tan80^\circ)^2$$ $$=\{\tan20^\circ+(-\tan40^\circ)+\tan80^\circ\}^2$$ $$-2\{\tan20^\circ(-\tan40^\circ)+\tan20^\circ\cdot\tan80^\circ+\tan80^\circ(-\tan40^\circ)\}$$

$$=(3\sqrt3)^2-2(-3)=33$$

[

Applying the following identities, $$\begin{align*} \cos 2A+\cos 2B&=2\cos(A-B)(A+B),\\ \sin2A&=2\sin A\cos A,\\ 2\cos A\cos B&=\cos(A-B)+\cos(A+B) \end{align*}$$
we get $$\begin{align*} \cos40^{\circ} + \cos80^{\circ} + \cos160^{\circ}&=0\\ \cos40^{\circ}\cos80^{\circ} + \cos80^{\circ}\cos160^{\circ} + \cos160^{\circ}\cos40^{\circ}&=-\frac{3}{4}\\ \end{align*}$$

$$\text{ and } \cos40^{\circ} \cos80^{\circ} \cos160^{\circ}=-\frac{1}{8}$$

Then the cubic equation whose roots are $\cos40^{\circ}, \cos80^{\circ}, \cos160^{\circ}$ is $$ x^3-\frac{3}{4}x+\frac{1}{8}=0 $$

]

In $(7)$ from this answer, it is shown that $$ \sum_{l=1}^n\tan^2\left(\frac{\pi l}{2n+1}\right)=n(2n+1) $$ In this case, $n=4$ and you're missing $l=3$. $\tan^2(60^\circ)=3$, so the sum would be $$ 36-3=33 $$

Notice that for $\theta = 20, 40,$ and $80$ degrees you have $\tan^2(3\theta) = 3$. The tangent triple angle formula, which you can get from the tangent angle addition formula, says that $$\tan(3\theta) = {3\tan(\theta) - \tan^3(\theta) \over 1 - 3 \tan^2(\theta)}$$ So the equation $\tan^2(3\theta) = 3$ can be expressed as $$(3\tan(\theta) - \tan^3(\theta))^2 = 3(1 - 3 \tan^2(\theta))^2$$ After a little algebra, this becomes the following, where $x = \tan(\theta)$. $$x^6 - 33x^4 + 27x^2 - 3 = 0$$ By the above, this has roots $x = \tan(20^\circ), \tan(40^\circ),$ and $\tan(80^\circ)$. Since $x$ only appears to even powers here, the other roots must be $x = -\tan(20^\circ), -\tan(40^\circ),$ and $-\tan(80^\circ)$. The sum of the squares of all six roots is thus given by $2(\tan^2(20^\circ) + \tan^2(40^\circ) + \tan^2(80^\circ))$. However, if we write these roots as $r_1,...,r_6$, then we also have $$\sum_i r_i^2 = \left(\sum_i r_i\right)^2 - 2\sum_{i < j} r_ir_j$$ But $\sum_i r_i$ is the coefficient of $x^5$ in the above equation, namely zero, and $\sum_{i < j} r_ir_j$ is the coefficient of $x^4$, namely $-33$. So you get $$2(\tan^2(20^\circ) + \tan^2(40^\circ) + \tan^2(80^\circ)) = -2\times-33$$ So we conclude that $$\tan^2(20^\circ) + \tan^2(40^\circ) + \tan^2(80^\circ) = 33$$