How to solve this recurrence relation using generating functions: $a_n = 10 a_{n-1}-25 a_{n-2} + 5^n\binom{n+2}2$?

Hint for finding the GF.

Note that $(n+1)(n+2)x^n=\frac{d^2}{dx^2}\left(x^{n+2}\right)$ and therefore $$\sum_{n=2}^\infty {n+2 \choose 2}5^nx^n = \frac{1}{2\cdot 5^2}\frac{d^2}{dx^2}\left( \sum_{n=2}^\infty (5x)^{n+2} \right).$$ Then recall that $\sum_{n=0}^{\infty}z^n=\frac{1}{1-z}$.

Moreover in your attempt it should be $$\sum_{n=2}^\infty a_n x^n = 10 \sum_{n=2}^\infty {a_{n-1}}x^n-25 \sum_{n=2}^\infty a_{n-2} x^n + \sum_{n=2}^\infty {n+2 \choose 2}5^nx^n.$$ that is $$\sum_{n=2}^\infty a_n x^n = 10x \sum_{n=1}^\infty {a_{n}}x^n-25x^2 \sum_{n=0}^\infty a_{n} x^n + \sum_{n=2}^\infty {n+2 \choose 2}5^nx^n.$$ Can you take it from here and find $f(x) =\sum_{n=0}^\infty a_n x^n$?

Alternative way for finding $a_n$ without the GF.

The given recurrence is a non-homogeneous linear recurrence relation with constant coefficients with characteristic equation $$z^2-10z+25=(z-5)^2=0$$ and non-homogeneous term $5^n {n+2 \choose 2}$ which is a second degree polynomial multiplied by a power of $5$, which is the root of multiplicity $2$ of the characteristic equation. Hence the general term of the recurrence with $a_0 = 1$, $a_1 = 15$ has the form $$a_n= 5^n(An^4+Bn^3+Cn+D)$$ where $A,B,C,D$ are real constant to be determined.

$\sum_{n=0}^\infty {n+2 \choose 2}5^nx^n = \frac12\sum_{n=0}^\infty(n^2+3n+2)(5x)^n=\frac12[\sum_{n=0}^\infty n^2y^n+3\sum_{n=0}^\infty ny^n+2\sum_{n=0}^\infty y^n]$

where $y=5x$. You will need the following series sums:

$(1)\sum_{n=0}^\infty n^2y^n=\dfrac{y(1+y)}{(1-y)^3}\\(2)\sum_{n=0}^\infty ny^n=\dfrac y{(y-1)^2}$