Groups under Multiplication

You're fourth step is a bit of an overkill, in terms of testing whether a subset is a subgroup. Indeed, we need to check:

• $(1)$ the identity element of $G$, namely $e = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$ is in $H$ and that
• $(2)$ for two elements $h_1, h_2 \in H$, $h_1\cdot h_2 \in H$,

But with respect to testing for inclusion of inverses in H (whether for every element $h \in H$, we also have that $h^{-1} \in H)$, we don't need to test for inverses of products of elements, since you've shown $H$ is closed under multiplication.

Indeed, you are correct that for most $h$ with integer entries $a, b$, the inverse $h^{-1} \notin H$ because the corresponding non-zero entries will likely be rational numbers, but not both integers.

So, taking for example $h = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}$, when we compute $h^{-1} = \begin{bmatrix} \frac 12 & 0 \\ 0 & \frac 12\end{bmatrix}$, we've shown that $h^{-1} \notin H$, because $\frac 12 \notin \mathbb Z$.

Hence, we've shown that $H$ is not closed under taking inverses. (All you need to provide is one counterexample of some element $h \in H$ such that $h^{-1} \notin H$, to show that $H$ is not closed under taking inverses.) And we therefore conclude that $H$, as defined, is NOT a subgroup of $G$. And then you're done.