Is there a geometric explanation for why principal curvature directions are orthogonal?

I only know that operator $df(X) \cdot dN(Y)$ is symmetric because the partial derivatives commute. The formal proof of that is just one line in W.P.A. Klingenberg "A Course in Differential Geometry", p.38. Can be this treated as a geometric intuition?

Informally, I would think of $df(X) \cdot dN(Y)$ as being defined completely just by by $df$ (and the dot product in the ambient space, which is $\mathbb{R}^3$ in the question), so the changes in direction $X$ are reflected automatically (via $N=\frac{df}{\left\Vert df\right\Vert }$) in direction $Y$. Maybe, this observation can be made more rigorous.

Edit. I've just noticed an error in the above. Actually, $N=\pm \frac{ds}{\left\Vert ds\right\Vert }$ for a defining function $s$ of the hypersurface.

My answer is really just an explanation of Qiaochu Yuan's comments.

Let's first take a look at how curvature is defined on 1-dimensional curve: suppose that $C$ is a twice continuously differentiable immersed plane curve represented by $\gamma (s)=(x(s),y(s))$, then curvature is defined as $\kappa (s)= \|T'(s)\|=\|\gamma{''}(s)\|$ ($T(s)$ the unit tangent vector and $s$ the arc length). We can see $\kappa(s)$ only depends up to second derivative.

Have this in mind, we are safe to replace any smooth surface with its quadratic approximation locally to find corresponding principal curvatures/directions. And you can easily check for any quadratic surface the principal directions are orthogonal.

The principal directions are orthogonal because they are the eigenvectors of a selfadjoint operator acting on the tangent plane, namely the Weingarten operator (shape operator) $W(v)$. This is defined by differentiating the normal vector in the direction of $v$, obtaining $W(v)$.