Show that if $a\neq b$, then we have for the $n\times n$-matrix $\textrm{det}=\frac{a^{n+1}-b^{n+1}}{a-b}$.

Let's call $A_n$ the matrix for which you want to compute the determinant and let's see how it can be constructed iteratively: $$ A_{n+1} = \begin{pmatrix} a+b & ab & 0 & \ldots & 0 & 0 \\ 1 & a+b & ab & \ldots & 0 & 0 \\ 0 & 1 & a+b & \ldots & 0 & 0 \\ \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \\ 0 & 0 & 0 & \ldots & a+b & ab \\ 0 & 0 & 0 & \ldots & 1 & a+b\end{pmatrix} = \begin{pmatrix} A_n & \begin{matrix} 0 \\ \cdots \\ 0 \\ ab \end{matrix} \\ \begin{matrix} 0 & \cdots & 0 & 1 \end{matrix} & a+b \end{pmatrix} $$

Such layout cries out for induction to compute the determinant $D_n = \det(A_n)$ . For $n=1$, we have $A_1 = (a+b)$ so $D_1 = a+b$ and $D_1 \cdot (a-b) = a^2 - b^2 $.

Now, if we assume the formula is true up to an integer $n$, can we prove that it is also true for $n+1$ ?

We can use Laplace expansion applied to the last column to find out that: $$ D_{n+1} = (a+b)\cdot D_n - ab \cdot D_{n-1}$$

Therefore: $$ \begin{align} (a-b)\cdot D_{n+1} & = (a+b)\cdot (a-b) D_n - ab \cdot(a-b) D_{n-1} \\ & = (a+b)\cdot( a^{n+1} - b^{n+1} ) - ab \cdot( a^{n} - b^{n} ) \\ & = a^{n+2}- ab^{n+1} + ba^{n+1} - b^{n+2} - ba^{n+1}+ ab^{n+1} \\ & = a^{n+2}- b^{n+2}\\ \end{align} $$

And we are done for the first question.

The second question asks what is happening when $a=b$. Obviously we can't apply the formula above since it is only valid for $a \ne b $. But we can take $b$ as close as we want to $a$, let's say $b_k = a + \frac{1}{k}$. Now we can apply the previous result, meaning that for all positive integer $k$: $$ \det \begin{pmatrix} a+(a + \frac{1}{k}) & a(a + \frac{1}{k}) & \ldots & 0 & 0 \\ 1 & a+(a + \frac{1}{k}) & \ldots & 0 & 0 \\ 0 & 1 & \ldots & 0 & 0 \\ \ldots & \ldots & \ldots & \ldots & \ldots \\ 0 & 0 & \ldots & a+(a + \frac{1}{k}) & a(a + \frac{1}{k}) \\ 0 & 0 & \ldots & 1 & a+(a + \frac{1}{k})\end{pmatrix} = \frac{\left(a + \frac{1}{k}\right)^n - a^n}{ \frac{1}{k} } $$

To conclude, you can use the continuity of the determinant and take the limit on both sides as $k \to \infty $. On the left you have the determinant of the matrix $A_n$ with $a=b$ and on the right you have the derivative of the function $ x \mapsto x^n $ at $x=a$, which is $n a^{n-1}$.

Let $L=\pmatrix{1\\ 1&\ddots\\ &\ddots&\ddots\\ &&1&1}$ and treat $a$ and $b$ as two indeterminates. Then the matrix in question is equal to $A=D^{-1}(aL+bL^T)D$, where $D=\operatorname{diag}(1,a,a^2,\ldots,a^{n-1})$. Since $\det L=1$, we get \begin{align} \det A&=\det(aL+bL^T) =\det\left(aI+bL^TL^{-1}\right)\\ &=(-b)^n\det\left(-\frac abI-L^TL^{-1}\right) =(-b)^nf\left(-\frac ab\right),\tag{1} \end{align} where $f(x)$ is the characteristic polynomial of $L^TL^{-1}$. Now, if you calculate $L^TL^{-1}$ directly, you will see that it is the transpose of a companion matrix. Hence you may read off the coefficients of $f$ directly from the entries of $L^TL^{-1}$. Then, using $(1)$, you should find that $$ \det A=(-b)^nf(-a/b)=a^n+a^{n-1}b+\cdots+ab^{n-1}+b^n.\tag{2} $$ Now the rest are straightforward when the indeterminates $a,b$ are specialised to two numbers.