Determine whether the following are groups or not...

Closures are trivial. Let's list what we have to prove:

i) Associativity;

ii) Existence of identity;

iii) Existence of inverses.

For $(\Bbb R, \ast)$, you should see what $(a \ast b) \ast c$ and $a \ast (b \ast c)$ are. What you have done does not makes it clear that they're equal. We have: $$\begin{align} (a \ast b) \ast c &= (a + b + ab)\ast c = a+b+ab+c + ac+bc+abc \\ a \ast (b \ast c) &= a \ast (b + c + bc) = a+b+c+bc+ab+ac+abc\end{align}$$ hence they're equal. So we've checked i). For the identity, we want $e \in G$ such that $a \ast e = e \ast a = e$ for all $a \in G$. So we want: $$a+e+ae= a \implies e + ae = 0 \implies e(1+a) = 0$$ Now, $e = 0$ will work for every $a$, but if $a = -1$, we have a problem here, because, calling $a^{-1}$ a candidate for the inverse of $-1$, we get: $-1 \ast a^{-1} = -1 + a^{-1} + (-1)a^{-1} = 0$, and so $-1 = 0$ by cancelling $a^{-1}$, a contradiction. Hence $-1$ doesn't have an inverse, and $(\Bbb R, \ast)$ is not a group. Property iii) fails.

Now, let's see the other one.

Consider $(\Bbb Z^2, \ast)$. If we want to see that $$\left((a,b)\ast(c,d)\right) \ast (q,r) = (a,b) \ast \left((c,d) \ast (q,r)\right)$$ then we should see each side separately, and then compare. We have: $$\begin{align} \left((a,b)\ast(c,d)\right)\ast (q,r) &= (ad+bc, bd)\ast(q,r) = (adr+bcr + bdq,bdr) \\ (a,b) \ast \left((c,d) \ast (q,r)\right) &= (a,b) \ast (cr + dq, dr) = (adr + bcr + bdq, bdr)\end{align}$$ hence property i) holds. For ii), we want $(e_1, e_2)$ such that $(a,b) \ast (e_1, e_2) = (e_1, e_2) \ast (a,b)$, that is: $(ae_2 + be_1, be_2) = (a,b)$, which gives $e_2 = 1$. And that gives $a + be_1 = a$, so $e_1 = 0$. To check that $(0,1)$ really is the identity, see that $(0,1)\ast(a,b) = (0b + a, 1b) = (a,b)$. So, property ii) also holds. Now, let $(\bar{a}, \bar{b})$ be a candidate for the inverse of $(a,b)$. We want: $(a,b) \ast (\bar{a}, \bar{b}) = (0,1)$. That is: $$(a\bar{b} + b\bar{a}, b\bar{b}) = (0,1)$$ This already gives us problems, since if you choose $b = 0$, the inverse will not exist. Hence $(\Bbb Z^2, \ast)$ is not a group, because property iii) fails.

For the first, note the identity must be $0$. However, then $-1$ does not have an inverse.

For the second, the identity must be $(0,1)$. However, then $(0, 0)$ has no inverse.

For the first problem, you should be careful about replacing $f$ with $a^{-1}$, since we don't even know if $a^{-1}$ even exists yet. Indeed, $a^{-1}$ is not the inverse of $a$ under addition; it's the inverse of $a$ under the operation that we're talking about, namely $*$. In fact, $(\mathbb R, *)$ is almost a group, if it weren't for the fact that $-1$ has no inverse.