Coset multiplication giving a well defined binary operation
The issue with working with quotient groups is that there are many representatives of the same coset. For example, in $\mathbb{Z}/5\mathbb{Z}$ one has that $$1+5\mathbb{Z}=\{\ldots,-9,-4,1,6,11,\ldots\}=6+5\mathbb{Z}$$ and $$2+5\mathbb{Z}=\{\ldots,-8,-3,2,7,12,\ldots\}=12+5\mathbb{Z}.$$ It is, of course, reasonable to be concerned whether $$3+5\mathbb{Z}=(1+5\mathbb{Z})+(2+5\mathbb{Z})=(6+5\mathbb{Z})+(12+5\mathbb{Z})=18+5\mathbb{Z}?$$ Of course, in this case everything works out just fine, but it is not always so. For example, take the subgroup $H=\langle(12)\rangle=\{(1),(12)\}\leq S_3$. We have $$(13)H=\{(13),(123)\}=(123)H$$ and $$(23)H=\{(23),(321)\}=(321)H$$ However, $(13)(23)H=(321)H$, while $(123)(321)H=(1)H=H$. Hence, $$(13)H(23)H=(13)(23)H=(321)H\neq H=(123)(321)H=(123)H(321)H$$ and the operation is not well defined.
The difference in the two cases is that $5\mathbb{Z}$ is a normal subgroup of $\mathbb{Z}$, while $H$ is not normal in $S_3$.
If we assume $H$ is a normal subgroup of $G$, we can show that the operation $aHbH=abH$ is well defined as follows:
Suppose $aH=cH$ and $bH=dH$. By definition, this means that $c^{-1}a\in H$ and $d^{-1}b\in H$. To show that $abH=cdH$, we need to show that $(cd)^{-1}(ab)\in H$.
Well, $$ (cd)^{-1}ab=d^{-1}c^{-1}ab=(d^{-1}(c^{-1}a)d)(d^{-1}b). $$ By assumption $d^{-1}b\in H$. Also, since $c^{-1}a\in H$ and $H$ is normal $d^{-1}(c^{-1}a)d\in H$. Finally, $H$ is a subgroup, so $(d^{-1}(c^{-1}a)d)(d^{-1}b)\in H$ and we're done.
Since $H$ is normal, $Hb=bH$ and so: $$aHbH=a(Hb)H=a(bH)H=abHH=abH$$
Given two cosets $aH, bH$, showing that the rule $(aH)(bH)=abH$ is well-defined amounts to showing that this product is independent of choice of coset representatives.
Let $a, a', b, b' \in G$ be such that $aH=a'H$ and $bH=b'H$.
We want to see that $(aH)(bH)=abH=a'b'H=(a'H)(b'H)$. It suffices to see $abH=a'b'H$.
Note first that $abH=ab'H$.
Since $aH=a'H$, there exists $h \in H$ such that $a=a'h$, so that $ab'H=a'hb'H$.
But $H$ is normal in $G$, so $(b')^{-1}hb'=h'$, for some $h' \in H$.
Then $hb'H=b'h'H=b'H$, so $abH=ab'H=a'hb'H=a'b'H$.