How many $k$-letter words are there such that the letters A and B are not next to each other?

You can handle this with coupled recurrences. Let $A(k)$ be the number of words without $AB$ or $BA$ that end in a character other than $A$ or $B$. Let $B(k)$ be the number of words without $AB$ or $BA$ that end in $A$ or $B$. You can add any character to an $A(k)$ word, but only $n-1$ characters to a $B(k)$ word, one of which leaves it as a $B(k+1)$ word. The recurrences are $$A(k+1)=(n-2)A(k)+(n-2)B(k)\\ B(k+1)=2A(k)+B(k)\\ A(0)=1\\ B(0)=0$$ For small $k$ you can make a spreadsheet to do these. For large $k$ you can do the eigenvalue/eigenvector thing on the matrix $$\begin {pmatrix} n-2&n-2\\2&1 \end {pmatrix}$$ The leading eigenvalue is $\frac 12\left(\sqrt{n^2+2n-7}+n-1\right)\approx n$ when both $k$ and $n$ are large. You want $A(k)+B(k)$ I made a spreadsheet for $n=10$, shown below. In the line for $n=2$ the $98$ under $A+B$ shows there are $98$ two character strings that are not $AB$ or $BA$. As there are $10^2=100$ unrestricted strings and we rule out $2$, this is correct. The $18\ B$ strings have one of $9$ characters (not a $B$) then an $A$, or one of $9$ characters (not an $A$) then a $B$.

Consider the following automaton that accepts the strings not containing AB or BA,

From the DFA, using Chomsky-Schutzemberger method, we get, $$q_0 = 1+xq_1+xq_2+(n-2)xq_0$$ $$q_1 = 1+xq_1+(n-2)xq_0$$ $$q_2 = 1+xq_2+(n-2)xq_0$$ (We omit $q_3$ and $q_4$ because it is a dead state)

On solving the above equations (i.e. substituting $q_1$ and $q_2$) we get the following expression for $q_0$, $$q_0 = \frac{1+x}{1-(n-1)x-(n-2)x^{2}}$$ This is our generating function. The coefficient of $x^{k}$ gives the number of strings of length k not having AB or BA. For example, for $n = k = 10$, Wolfram Alpha gives the following Taylor Series,

And as you can see, the coefficient of $x^{10}$ is equal to the answer you have provided.