How can you mathematically define a "wobbly" function?

Perhaps you're thinking of total variation. You can think of it as "vertical distance traveled". You could, perhaps, over some region, divide the range by the total variation. That is, you could define your "wobbliness" as $$W_{[a,b]}(f)=\frac{\displaystyle\sup_{x\in[a,b]}f(x)-\inf_{x\in[a,b]}f(x)}{V_{\!b}^{\!a}(f)}, $$ where, of course, we would have to have $V_{\!b}^{\!a}(f)\not=0.$ This $W_{[a,b]}(f)$ would be a number in the interval $(0,1],$ and it would be undefined for a constant function. You could define the "wobbliness" of a constant function to be one. $W_{[a,b]}(f)$ would be close to zero if the function was very wobbly. It would equal $1$ if the function was monotonic. This idea has the virtue of being well-defined even for not-very-well-behaved functions.

If the function is differentiable, then I would use arc length as follows: $$W_{[a,b]}(f)=\frac{\sqrt{(b-a)^2+(f(b)-f(a))^2}}{\int_a^b\sqrt{1+(f'(x))^2}\,dx}.$$ This would be near zero for very wobbly functions, and one for a straight line. No issue with divide-by-zero in this case.

You can use the curvature of the graph to measure how non-straight the graph is at a given point. It is given by $$ \kappa(x) = \frac{|f''(x)|}{\big(1+(f'(x))^2\big)^\frac32}$$ It's similar to your first guess $|f''(x)|$ but the denominator causes it to become small if the slope is big, because in this situation even big $f''(x)$ does not mean a big change in the direction of the graph.

The integral of the curvature over an arc length (not over just $x$) has an interesting property, that it measures the angle by which the graph has turned on a giving interval:

\begin{align} K_{[a,b]} =\int_a^b \kappa (x) ds(x) &= \int_a^b \frac{|f''(x)|}{\big(1+(f'(x))^2\big)^\frac32} \sqrt{1+(f'(x))^2} dx = \\ &= \int_a^b \frac{|f''(x)|}{1+(f'(x))^2} dx = \\ &= \int_a^b \left| \frac{d}{dx} \arctan f'(x)\right| dx = \\ &= \int_a^b \left| \frac{d}{dx} \varphi(x)\right| dx\end{align} where $\varphi(x)$ denotes the declination angle of the graph at point $x$. The absolute value makes sure that turns left and turns right acccumulate, and not cancel each other. As a consequence $K_{[a,b]}$ does not depend on the scale: if you stretch the graph proportionally in all directions, it remains the same. It is also invariant under isometries (translations, rotations). It is equal $0$ on intervals where function graph is straight, otherwise always greater that $0$, it can be used even for generic curves (that are not a graph of a function), although then the curvature needs to be calculated from a different formula. It is also extensive, that is for $a<b<c$ we have $K_{[a,c]}= K_{[a,b]}+K_{[b,c]}$.

In the comments @AdrianKeister proposed another scale-invariant quantity, the ratio of arc length and the straight-line distance:

$$ R_{[a,b]} =\frac{\int_a^b \sqrt{1+(f'(x))^2} dx}{\sqrt{(b-a)^2+(f(b)-f(a))^2}}$$

It is equal to $1$ for straight lines, otherwise always greater than $1$. It is scale-invariant, and invariant under isometries. It seems to work well-enough for graphs of functions, although for other curves it can exhibit some strange behavior; for example, for a closed curve (like a full circle) you have $R_{[a,b]} = \infty$, then it becomes finite again if I take more than a full circle. It can be made arbitrarily bigg just by stretching the graph in the $y$ direction ($f(x) \rightarrow \lambda f(x)$). It's not extensive, and actually adding sometimes considering a bigger interval can make $R$ smaller.