Normalizer of group action

Suppose $g\in G$ stabilizes the orbit space $H\,\backslash X$ (that is, the collection of orbits of $H\curvearrowright X$).

Since $g$ cannot mix the $G$-orbits, it must stabilize each orbit space $H\,\backslash \mathcal{O}_1$, $\cdots$, $H\,\backslash \mathcal{O}_k$ (where $\mathcal{O}_i$ are the orbits of the full action $G\curvearrowright X$). Any $G$-orbit is isomorphic (as a $G$-set) to $G/K$, where $K$ is the stabilizer of some point in the orbit, so we ought to examine the situation for $G/K$.

Note that the $H$-orbit space $H\,\backslash(G/K)$ is the set $H\,\backslash G/K$ of double cosets. Also left and right actions can be converted using inverses, and in particular there is an action $H\times K\curvearrowright G$ given by $(h,k)g:=hgk^{-1}$ and the orbits are precisely the double cosets; $(H\times K)\,\backslash G=H\,\backslash G/K$. In particular, the double cosets partition the group $G$, just as orbits partition any $G$-set.

For $g$ to stabilize $H\,\backslash G/K$, we must have for all $a\in G$ there exists a $b\in G$ such that $gHaK=HbK$. As every element of an orbit is a representative, and $ga\in gHaK$, we have $gHaK=HgaK$. This is equivalent to saying $H({}^cK)=(H^g)({}^cK)$ for all $c\in G$, where $H^g=g^{-1}Hg$ and ${}^cK=cKc^{-1}$ and $c=ga$. Thus,

$$ \mathrm{Stab}_G(H\,\backslash G/K)=\{g\mid H({}^cK)=(H^g)({}^cK)~\forall c\in G\}. $$

I am not sure if this simplifies any better. Then $\mathrm{Stab}_G(H\,\backslash X)$ will be the intersection of the individual stabilizers $\mathrm{Stab}_G(H\,\backslash \mathcal{O}_i)$. Two extreme situations:

• $\mathrm{Stab}_G(H\,\backslash G/G)=G$
• $\mathrm{Stab}_G(H\,\backslash G/1)=N_G(H)$.

In general, $\mathrm{Stab}_G(H\,\backslash X)$ will be between $N_G(H)$ and $G$.

I wanted to extend the analysis of runway44's answer to the case where $G$ acts transitively on $X$, and give an example that shows that even here, the answer is negative.

If we assume that $G$ acts faithfully and transitively on $X$, then we can put $K = \mathrm{Stab}_{G}(x)$ for some arbitrary $x \in X$; then there is a natural correspondence between $X$ and $G/K$, the cosets of $K$ (in contrast to runway44's answer, I am going to be working with right cosets, and a right group action). So each element of $X$ corresponds to a coset $Kg$ for some $g \in G$, and an $H$-orbit on $X$ corresponds to the double-coset $KgH$.

Now, if some $n \in G$ stabilizes the $H$-orbits on $X$, then for each $g \in G$ we have $KgHn = Kg^{\prime}H$ for some $g^{\prime} \in G$. Since $gn \in KgHn$, we have $KgHn = KgnH$. On the other hand, we can write $KgHn = Kgn(n^{-1}Hn)$ giving us $KgnH = Kgn(n^{-1}Hn)$; since $gn$ covers all the elements of $G$ as $g$ does, this means $$KgH = Kg(n^{-1}Hn)$$ for all $g \in G$, or equivalently, $$x^{H} = x^{n^{-1}Hn}$$ for all $x \in X$. In other words, the most we can conclude is that $H$ and $n^{-1}Hn$ have the same orbits on $X$.

From this, we can see that even when $G$ acts faithfully and transitively on $X$, it is possible that $N_{G}(H) < \mathrm{Stab}_{G}(O(H))$. For example, let $G = S_{6}$ acting naturally on $\{1,2,3,4,5,6\}$, and take $H = \langle (1,2,3)(4,5,6),\ (1,2) \rangle$. If $n = (1,4)(2,5)(3,6)$, then $n$ stabilizes the orbits of $H$. But $$n^{-1}Hn = \langle (1,2,3)(4,5,6),\ (4,5)\rangle \ne H.$$