Isomorphism between the actions of $G$ on $G/H$ and $G/gHg^{-1}$?

We want a bijection that is well-defined. If $xH=yH$, then $y^{-1}x\in H$, hence $gy^{-1}xg^{-1}\in gHg^{-1}$, hence $yg^{-1}(gHg^{-1}) = xg^{-1}(gHg^{-1})$.

So, how about mapping the coset $aH$ to the coset $ag^{-1}(gHg^{-1})$?

Why that and not try something like “hence $gy^{-1}g^{-1}gxg^{-1}\in gHg^{-1}$, hence $gxg^{-1}(gHg^{-1}) = gyg^{-1}(gHg^{-1})$; so let’s define the image of $aH$ to be $gag^{-1}(gHg^{-1})$”, which was your second attempt? Because that extra $g$ on the left interferes with the action of $G$, whereas mapping $aH$ to $ag^{—1}(gHg^{-1})$ won’t run into that problem.

Alternatively, you can recognize the map $aH\mapsto ag^{-1}(gHg^{-1})$ as the result of your second map followed by “multiply by $g^{-1}$ on the left”. Essentially, it partially does the conjugation you want to do, but codes it into the bijection instead of into the action.