Showing $b^2 \notin \langle a \rangle$ for $V_{8n}=\langle a,b:a^{2n}=e, b^4 = e, ba=a^{-1}b^{-1},b^{-1}a=a^{-1}b\rangle.$

From the relations $bab=a^{-1}$ and $ba^{-1}b = a$, we get $b^2ab^2 = a$, and since $b^4=1$, this implies $b^{-2}ab^2 = a$, so $b^2 \in C_G(a)$. Since clearly $b^2 \in C_G(b)$, this gives $b^2 \in Z(V_{8n})$.

Now, the calculation in user1729's answer shows that $V_{8n}/\langle b^2 \rangle$ is isomorphic to the fihedral group of order $4n$. Since $b^4=1$, this proves that $|V_{8n}| \le 8n$ and $|V_{8n}| \ge 4n$. But to prove that $b^2 \not\in \langle a \rangle$, we still have to prove that $b^2 \ne 1$, which would show that $|V_{8n}| = 8n$.

To prove that, we follow the standard procedure of contructing a group $\langle a,b \rangle$ of order $8n$ in which $a$ and $b$ satisfy the relations of $V_{8n}$.

Start with the group $H = \langle a,c \mid a^{2n}=c^2=1, ac=ca \rangle \cong C_{2n} \times C_2$. Then $H$ has an automorphism $\phi$ of order $2$ with $\phi(a) = a^{-1}c$ and $\phi(c)=c$. Let $\langle b \rangle$ with $b$ of order $4$ be cyclic, and let $X$ be the semidirect product $H \rtimes_\phi \langle b \rangle$. So $$X = \langle a,b,c \mid a^{2n}=b^4=c^2=1, ac=ca, b^{-1}ab=a^{-1}c,b^{-1}cb=c \rangle,$$ and $|X| = 4|H| = 16n$.

Now $b^2, c \in Z(X)$, and we define $G = X/\langle b^2c^{-1} \rangle$, so $|G| = 8n$ and $$G = \langle a,b,c \mid a^{2n}=b^4=c^2=1, b^2=c, ac=ca, b^{-1}ab=a^{-1}c,b^{-1}cb=c \rangle \cong \langle a,b\mid a^{2n}=b^4=1, ab^2=b^2a, b^{-1}ab=a^{-1}b^2 \rangle.$$

Note that in $G$ we have $bab = b^2a^{-1}b^2 = a^{-1}$ (since $b^2 \in Z(G))$, and similarly $ba^{-1}b = a$, so the relations of $V_{8n}$ are indeed satisfied in $G$, and since $|G| = 8n$, this proves that $|V_{8n}| = 8n$ and $G \cong V_{8n}$.

Finally note that, in $G$, $b^2 = c \not\in \langle a \rangle$, which is what you wanted to prove.

This answer uses semidirect products. I hope you are familiar with them, else see the wikipedia link :-)

The following proves that if $b^2\in\langle a\rangle$ then $b^2=1$ (therefore it is an incomplete answer - but some of the working is applied in Derek Holt's answer, so I'll leave it here). One way to see this partial result is to quotient out $b^2$ from the group (that is, quotient out the normal subgroup generated by $b^2$, which I denote by $\langle\langle b^2\rangle\rangle$): $$\begin{align*} V_{8n}/\langle\langle b^2\rangle\rangle &= \langle a, b : a^{2n} = e, b^4 =e, ba = a^{-1}b^{-1}, b^{-1}a = a^{-1}b \rangle/\langle\langle b^2\rangle\rangle\\ &= \langle a, b : a^{2n} = e, b^4 =e, ba = a^{-1}b^{-1}, b^{-1}a = a^{-1}b, b^2 \rangle\\ &= \langle a, b : a^{2n} = e, b^2 =e, bab^{-1} = a^{-1}, bab^{-1} = a^{-1} \rangle\\ &= \langle a, b : a^{2n} = e, b^2 =e, bab^{-1} = a^{-1}\rangle \end{align*} $$ This is a presentation for the semidirect product of $\mathbb{Z}_{2n}\rtimes\mathbb{Z}_2$. In particular, the image of $\langle a\rangle$ has order $2n$ in the quotient, and so $a$ has the same order both in $V_{8n}$ and in the quotient group. Therefore, the subgroup $\langle a\rangle$ intersects the kernel trivially, $\langle a\rangle\cap\langle\langle b^2\rangle\rangle=_{V_{8n}}1$, and hence if $b^2\in \langle a\rangle$ then $b^2=1$, as required.