A Particular Two-Variable System in a Group

In fact, it is true that the presentation $$\langle a,b\ |\ a^{-1}b^na=b^{n+1}, b^{-1}a^nb=a^{n+1}\rangle$$ always defines the trivial group.

Here's a proof: Let $M=n^{n+1}$, $N=(n+1)^{n+1}$, and check that we have $b^{-(n+1)}a^Mb^{(n+1)}=a^N$.

Now we also know from the relations that $ab^{(n+1)}=b^na$ and similarly $b^{-(n+1)}a^{-1}=a^{-1}b^{-n}$. So we also have

$$\begin{align} a^N&=b^{-(n+1)}a^Mb^{(n+1)} \\ &=(b^{-(n+1)}a^{-1})a^M(ab^{(n+1)})\\ &=a^{-1}b^{-n}a^Mb^na. \end{align}$$

Thus $a^N=b^{-n}a^Mb^n=a^K$, where $K=n\cdot (n+1)^n$. So we have $1=a^{N-K}=a^L$, where $L=(n+1)^n$. But if $P=n^n$ (sorry for all the letters!), we have $b^{-n}a^Pb^n=a^L=1$, so $a^P=1$.

But $\gcd(P,L)=\gcd(n,n+1)=1$, so $a=1$, and of course this implies $b^n=b^{n+1}$, so also $b=1$.

As it happens I have my copy of Baumslag & Chandler on the shelf. This exercise is listed as very hard. I seem to have pencilled in a solution 27-28 years ago. The text is really worn out, so I can only make out the first few steps. Damn, I really need a prescription for new glasses... Anyway here are the first three consequences of those relations: $$ a^{-1}b^8a=b^{12}, a^{-1}b^{12}a=b^{18}, a^{-1}b^{18}a=b^{27}. $$ As consequences of these I seem to have derived (you must rederive these for full credit) the following: $$ a^{-2}b^{12}a^2=b^{27}=a^{-3}b^8a^3=b^{-1}a^{-2}b^8a^2b. $$ The next consequence seems to be $a^{-2}b^8a^2=a^{-2}b^{12}a^2.$ From that point on the text is too blurred, but I think I might be able to redo that even though my brain has lost most of its agility over the years. Just in case this is homework I will stop here with these hints.

In this answer I give credit to Jyrki Lahtonen for the answer he posted.

There are holes in his post, so I sensed the need for a step by step answer (firstly to convince myself, but also other people in doubt), and so here it is.

$\bbox[5px,border:2px solid]{\begin{array}{cc}a^3=b^{-1}a^2b&(\alpha)\\b^3=a^{-1}b^2a &(\beta)\end{array}}$

$b^{6}=b^3b^3=(a^{-1}b^2a)(a^{-1}b^2a)=a^{-1}b^4a\quad(\alpha)$

$b^{12}=b^6b^6=(a^{-1}b^4a)(a^{-1}b^4a)=a^{-1}b^8a\quad(\alpha)$

$b^{18}=b^{12}b^6=(a^{-1}b^8a)(a^{-1}b^4a)=a^{-1}b^{12}a\quad(\alpha)$

$b^{27}=b^{18}b^6b^3=(a^{-1}b^{12}a)(a^{-1}b^4a)(a^{-1}b^2a)=a^{-1}b^{18}a\quad(\alpha)$

$b^{27}=a^{-1}b^{18}a=a^{-2}b^{12}a^2=a^{-3}b^8a^3=\ (\beta)\ =(b^{-1}a^2b)^{-1}b^8(b^{-1}a^2b)=(b^{-1}a^{-2}b)b^8(b^{-1}b^{-1}a^2b)=b^{-1}a^{-2}b^8a^2b=b^{-1}(a^{-1}(a^{-1}b^8a)a)b=b^{-1}(b^{18})b=b^{18}$

$b^{27}=b^{18}\Rightarrow b^9=1$

$b^9=b^3b^3b^3=(a^{-1}b^2a)(a^{-1}b^2a)(a^{-1}b^2a)=(a^{-1}b^6a)=1\Rightarrow b^6=1\quad (\beta)$

$b^3=b^9b^{-6}=1=a^{-1}b^2a\Rightarrow b^2=1$

$b=b^3b^{-2}=1$ and then $a^3=b^{-1}a^2b=a^2\Rightarrow a=1\quad (\alpha)$.

$\bbox[5px,border:2px solid]{a=b=1}$