Help me out with the sum $\sum_{n= 0}^{N-1} \frac{ \left(a-b \cos{\left(\frac{2 \pi n}{N} \right)} \right)^2}{a^2 + b^2 -2ab\cos{\frac{2\pi n}{N}}}$

I have to thank Vepir for the numerical work, it was immensely helpful in finding a formal derivation.

Let's look at an expression of the form $$ w=a-b\,e^{i\phi} $$ for which $|w|^2=(a-b\,e^{i\phi})(a-b\,e^{-i\phi})=a^2+b^2-2ab\cos\phi$ and $\Re(w)=a-b\cos\phi$. So we have: $$ \frac{(a-b\cos\phi)^2}{a^2+b^2-2ab\cos\phi}=\frac{(\Re(w))^2}{|w|^2}=\frac{1}{4}\frac{(w+w^*)^2}{|w|^2}=\frac{1}{4}\left(\frac{w^2}{|w|^2}+\frac{w^{*2}}{|w|^2}+2\right)=(\star) $$ Since $w^*/|w|^2=1/w$, we get $$ (\star)=\frac{1}{2}+\frac{1}{4}\left(\frac{w}{w^*}+\frac{w^*}{w}\right)=\frac{1}{2}\left(1+\Re\left(\frac{w}{w^*}\right)\right) $$ If we denote $\phi_n=\frac{2\pi n}{N}$, your sum can be rewritten as $$ \sum_{n=0}^{N-1}\frac{(a-b\cos\phi_n)^2}{a^2+b^2-2ab\cos\phi_n}=\frac{N}{2}+\frac{1}{2}\Re\sum_{n=0}^{N-1}\frac{a-b\,e^{i\phi_n}}{a-b\,e^{-i\phi_n}} $$ Notice that the numbers $b\,e^{-i\phi_n}-a$ are precisely the roots of the polynomial $(z+a)^N-b^N$ and since the product of the roots of a polynomial whose leading coefficient is $1$ is just its constant factor (up to sign), we get $$ \prod_{n=0}^{N-1}(b\,e^{-i\phi_n}-a)=(-1)^N(a^N-b^N)\\ \prod_{n=0}^{N-1}(a-b\,e^{-i\phi_n})=a^N-b^N $$ Why is this useful? Well, we can now calculate the following: $$ \frac{a-b\,e^{i\phi_n}}{a-b\,e^{-i\phi_n}}=\frac{(a-b\,e^{i\phi_n})\prod_{m\neq n}(a-b\,e^{-i\phi_m})}{\prod_{m=0}^{N-1}(a-b\,e^{-i\phi_m})}=\frac{(a-b\,e^{i\phi_n})\prod_{m\neq n}(a-b\,e^{-i\phi_m})}{a^N-b^N} $$ Thus we "only" have to evaluate $$ \sum_{n=0}^{N-1}(a-b\,e^{i\phi_n})\prod_{m\neq n}(a-b\,e^{-i\phi_m})=(\star\star) $$ Now there's some fiddling. Let's look at this again (using the formula for a difference of powers, $x^r-y^r=...$):$$ \prod_{m=0}^{N-1}(a-b\,e^{-i\phi_m})=a^N-b^N=a^N-(b\,e^{-i\phi_n})^N=(a-b\,e^{-i\phi_n})\sum_{k=0}^{N-1}a^{N-1-k}(b\,e^{-i\phi_n})^k\\ \prod_{m\neq n}(a-b\,e^{-i\phi_n})=\sum_{k=0}^{N-1}a^{N-1-k}b^k\,e^{-ik\phi_n} $$ So: $$ (\star\star)=\sum_{n=0}^{N-1}(a-b\,e^{i\phi_n})\sum_{k=0}^{N-1}a^{N-1-k}b^k\,e^{-ik\phi_n}=\sum_{n,k=0}^{N-1}\left(a^{N-k}b^k\,e^{-ik\phi_n}-a^{N-1-k}b^{k+1}\,e^{-i(k-1)\phi_n}\right) $$ But there are geometric series hiding inside there, so we switch the summation to $n$ and deal with the first part: $$ \sum_{k=0}^{N-1}a^{N-k}b^k\sum_{n=0}^{N-1}e^{-ik\phi_n}=\sum_{k=0}^{N-1}a^{N-k}b^k\cdot N\delta_{k,0}=Na^N $$ The $\delta$ is the Kronecker delta, I didn't bother writing out the computation explicitly since it's just a geometric series.
The exact same procedure on the second part gives $$ \sum_{n,k=0}^{N-1}a^{N-1-k}b^{k+1}\,e^{-i(k-1)\phi_n}=\sum_{k=0}^{N-1}a^{N-1-k}b^{k+1}\cdot N\delta_{k,1}=Na^{N-2}b^2 $$ Finally, we get $$ \sum_{n=0}^{N-1}(a-b\,e^{i\phi_n})\prod_{m\neq n}(a-b\,e^{-i\phi_m})=Na^N-Na^{N-2}b^2=Na^{N-2}(a-b)(a+b) $$ so that your sum equals $$ \sum_{n=0}^{N-1}\frac{(a-b\cos\phi_n)^2}{a^2+b^2-2ab\cos\phi_n}=\frac{N}{2}\left(1+\frac{a^{N-2}(a-b)(a+b)}{a^N-b^N}\right) $$ as claimed by Vepir.

It appears to me that your sum is given by:

$$ \sum_{n= 0}^{N-1} \frac{ \left(a-b \cos{\left(\frac{2 \pi n}{N} \right)} \right)^2}{a^2 + b^2 -2ab\cos{\frac{2\pi n}{N}}} = \frac{N}{2} \left(1+\frac{a^{N-2} (a - b) (a + b)}{a^N - b^N}\right) $$

This was obtained by Mathematica for some individual fixed $N$ at a time.

I'm not sure if it can show it holds for all $N$.

I'm not an expert in Mathematica. Maybe someone can utilize it better and verify this for all $N$.

It can prove this is true for $N=2,3,4,5,6,8,12,\dots$ for example, but I'm not sure why it can't simplify some other $N$ cases like $N=7$. The others can be tested numerically:

You can verify absolute differences between LHS and RHS using:

ClearAll[mySum, m, a, b, inputSum];
inputSum [m_, a_: a, b_: b] := Sum[(a - b Cos[(2 Pi n) / m])^2/(a^2 + b^2 - 2 a b  Cos[(2 Pi n)/m]), {n, 0, m - 1}];
mySum[m_, a_: a, b_: b] := 1/2 (1 + (a^(-2 + m) (a - b) (a + b))/(a^m - b^m)) m;

a = 20;
b = 10;
Do[Print[m, " ", DecimalForm[Abs[N[mySum[m, a, b] - inputSum[m, a, b]]], 100]], {m, 2, 100}]

By setting a,b to some fixed values. (Here I used m for $N$.) Also note that some results will be represented as 0.00000000....456 for example, due to precision setting of N[] function.