Prove that $\sqrt{11}-1$ is irrational by contradiction

No, you cannot conclude that $a^2$ is a multiple of $11$. You can instead rewrite $$ a^2=10b^2-2ab=2(5b^2-ab) $$ so $2\mid a$. Write $a=2c$, with $c$ integer. Then $$ 4c^2=2(5b^2-2bc) $$ or $5b^2=2(c^2+bc)$. Since $2\nmid 5$, we conclude $2\mid b$.

This is a contradiction to $a$ and $b$ being coprime.

we prove $\sqrt11$ is irrational by contradiction.

let $\sqrt11$ is rational,$\exists a,b$ such that $(a,b)=1$ and $\sqrt{11}=\frac{a}{b}$

$\implies 11=\frac{a^2}{b^2}$

$a^2=11.b^2 \implies 11\mid a^2 $ and $11$ is prime

therefore, $11\mid a, \exists a_1$ such that $a=11a_1$

$$a^2=(11a_1)^2=121a_1^2=11b^2$$ $$\implies b^2=11a_1^2 $$

$\implies 11\mid b$

it violated $(a,b)=1$.

by contradiction, $\sqrt{11}$ is irrational.

now I go to this problem, to prove $\sqrt{11}-1$ is irrational by contradiction,

assume that $\sqrt{11}-1$ is rational. We know that $1$ is rational number.

We also know that sum of two rational numbers is always rational number.

$[\sqrt{11}-1]+1$ is rational number.
but, $\sqrt{11}$ is irrational

by contradiction, $\sqrt{11}-1$ is irrational

Continuing where you stopped: $$ 10b^2 = a^2 + 2ab $$ Write this as $$ b(10b-2a) = a^2 $$ Therefore, $b$ divides $a^2$. Since $\gcd(a,b)=1$, the only possibility is $b=1$. But then $\sqrt{11}-1=a$ is an integer, which implies $\sqrt{11}$ is an integer, which it clearly isn't because $3^2 < 11 < 4^2$.