Let $H$ be a subgroup of $G$, and suppose that $G$ acts by multiplication over the set $X:=G/H$ of the left-hand side classes of $H$ over $G$.
The kernel of $\lambda$ should be $\bigcap_{g\in G}gHg^{-1}$. This is called the normal core of $H$.
I think this is what you want. $A$ doesn't have a kernel, as $X$ is only a set. But $\operatorname{Sym}X$ is a group, and $\lambda$ a homomorphism. So we can talk about $\operatorname{ker}\lambda:=\{g\in G:\lambda(g)=e\}$.