Intersection of all normalizers

No. There are non-abelian groups $G$ for which all subgroups are normal, such as the quaternion group of order 8. So the intersection of all normalizers is just $G$.

The intersection of all normalizers of subgroups of a group $G$ is called the norm of $G$. By a result of Schenkman [E. Schenkman, On the norm of a group, Illinois J. Math., 7 (1960) 150-152] the norm of a group always lies in $Z_2(G)$ of the upper central series. So the question has positive answer if for example the center $Z(G)=Z_2(G)$.

The Schenkman's result has been improved by Cooper [C.D.H. Cooper, Power automorphisms of a group, Math. Z., 107 (1968) 335-356.] as follows: an automorphism which leaves every subgroup of a group $G$ invariant induces the trivial automorphism in the central factor group $G/Z(G)$.

As was pointed out by Rickard and Weidner, the existence of Hamiltonian groups like $Q_8$ shows that the answer to the original question is "no". But all Hamiltonian groups have even order, so this leaves open the question for odd order groups. The answer is "no" for these groups too.

Let $p$ be an odd prime, and let $G$ be the unique nonabelian group of order $p^3$ and exponent $p^2$. Then $Z(G)$ has order $p$, but $G$ has an elementary abelian subgroup $E$ of order $p^2$, consisting of all elements $x \in G$ with $x^p = 1$. I claim that $E$ normalizes every subgroup of $G$. To see this, let $H \subseteq G$. If $|H| \geq p^2$ then $H$ is normal in the whole group $G$, and otherwise $H \subseteq E$, and since $E$ is abelian, $E$ normalizes $H$.