Which coefficients of the characteristic polynomial of the shape operator are isometric invariants?

If $n>2$ then all the coefficients $a_m$ are isometry invariants of generic $M$. Say, it sufficient to assume that $M$ has positive curvature, but as you will see much weaker assumptions can be made.

Let $e_i$ be the principle basis and $\kappa_i$ be principle curvatures. The curvature operator of $M$ has eigenvectors $e_i\wedge e_j$ with eigenvalues $K_{ij}=\kappa_i\cdot\kappa_j$. If $K_{ij}>0$ and $n>2$, you can recover $\kappa_i$ from $K_{ij}$, say $$\kappa_i=\sqrt{\frac{K_{ij}\cdot K_{ik}}{K_{jk}}}.$$ Since the values $K_{ij}$ are isometry invariants of $M$ so are all $\kappa_i$ and symmetrizing we get all $a_m$ (up to sign).

About other conditions: say nonzero Gaussian curvature $G=\kappa_1\cdots\kappa_n$ will do as well and the proof is the same. In particular $|G|$ is invariant.

On the other hand for the hyperplane $|H|=\mathrm{const}\cdot |a_1|$ is not invariant, so you should assume something about curvature at the point.