What does "the average continuous function is nowhere monotonic" mean?

He probably meant that either:

1. The space of continuous functions which are nowhere monotonic has probability one using Wiener measure. I.e. a continuous function is "almost surely" nowhere monotonic using the standard probability measure for that space.

2. That set of nowhere monotonic continuous functions is of the first category (Baire category theorem) with respect to the sup topology/ topology of uniform convergence.

Also see this question for a discussion of the definition of "nowhere monotonic".

Nowhere monotonic continuous function

Here is one possible interpretation of what he meant: every continuous, real-valued function on an interval can be approximated as well as you like by continuous, nowhere monotonic functions. In other words, these functions are dense.

In more detail, consider the space $C([0,1])$ (for example) of continuous functions $[0,1]\to\mathbb R$. To talk about the nowhere monotonic functions being dense, the space $C([0,1])$ needs to have some sort of topology, or better yet, a metric. Given $f,g\in C([0,1])$, say that the distance $d(f,g)$ between $f$ and $g$ is $$ d(f,g) = \sup_{t\in[0,1]} \big|f(t) - g(t)\big|. $$ (This is a metric!)

Theorem: Under this metric, the space of continuous, nowhere monotonic functions is a dense subspace of $C([0,1])$.

In other words, I can approximate continuous functions on the interval by nowhere monotonic functions: if you give me some random continuous function $f:[0,1]\to\mathbb R$, and some tolerance $\varepsilon>0$, I can find a nowhere monotonic function $g:[0,1]\to\mathbb R$ such that $|f(t)-g(t)|<\varepsilon$ for all $t\in[0,1]$.

This theorem has an easy proof using the Baire Category Theorem. Your professor can probably give it, or one of us at MSE can if you would like.

Denseness is only one notion of a behavior being "average", and it's not always the best one. For instance, the rationals are a dense subset of the reals even though the "average" real number is irrational. Several other answers give alternative notions of average.

One way to understand this is that nowhere monotonic functions form a "prevalent" set in the space of all continuous functions on, say, an interval, in the sense of Hunt, Sauer and Yorke.