Show tensor $g_{ij}A^iB^j $ is invariant under coordinate change

You just have to show that if the coordinate transform as $x^i \rightarrow \bar{x}^{\mu}$, then $$ \bar{g}_{\mu\nu} \bar{A}^{\mu}\bar{B}^{\nu} = g_{ij} A^{i} B^{j} $$ You can show this by first note that the components of the tensors in this new coordinate system is (by the transformation rule for tensor components) $$ \bar{g}_{\mu\nu} = g_{ij} \frac{\partial x^i}{\partial \bar{x}^{\mu} } \frac{\partial x^j}{\partial \bar{x}^{\nu} }$$ $$ \bar{A}^{\mu} =A^{k} \frac{\partial \bar{x}^{\mu} }{\partial x^k} \qquad \bar{B}^{\nu} =B^{l} \frac{\partial \bar{x}^{\nu} }{\partial x^l} $$ And after substitution to $ \bar{g}_{\mu\nu} \bar{A}^{\mu}\bar{B}^{\nu} $ you can use your hint above to show that it is equal to $g_{ij} A^{i} B^{j}$.

Here is an index-free solution. Represent $G=(g_{ij})$ as a matrix, $a=(A^i)$ and $b=(B^j)$ as vectors. Then $g_{ij}A^iB^j$ can be written as $a^\top Gb$. We change the coordinate system via a transformation $T$. This yields:

$$a\mapsto Ta,\qquad b\mapsto Tb,\qquad G\mapsto (T^{-1})^\top GT^{-1}.$$

We then have

$$a^\top Gb \quad\mapsto\quad (T a)^\top (T^{-1})^\top GT^{-1} (T b) = a^\top \underbrace{T^\top (T^{-1})^\top}_IG\underbrace{T^{-1} T}_I b = a^\top G b. $$

You can try writing this out with indices if necessary.