Examples where an ill-behaved function leads to surprising results?

I think, the most transparent example is phase transition: by definition it is when some thermodynamic value does not behave well.

AFAIK when Fourier showed that non-continuous function may be presented as an infinite sum of continuous, he had a hard time convincing people around that he is not crazy. That story might partially answer your question: as long as any not-so-well-behaved function may be presented as a sum of smooth ones, there is no much difference as long as good formulated laws are linear. Functions which are really bad behaved usually do not appear in real problems. If they do, there is some significant physics behind it (as with phase transition, shock wave, etc.) and one can not miss it.

For an operator it is better (for physicist) to think of function from operator as a function acting on its eigenvalues (if it is not diagonalizable, in physics it is bad behaviour). This is equivalent to power series definition, but works for any function.

I have had a surprizing result due to the wave function having different left and right derivatives at a point (see Chapter 2.1 and Appendix 3). Generally this article contains more surprizing results just due to implicit assumptions being wrong.

Well, I don't know if you want to count that, but QFT is full with functions that have poles which I'd call not well-behaved, and it does have lots of physical effects. If you're talking about observables only, you can approximate any discontinuous function to arbitrary precision with a continuous function, and you can push the difference below measurement precision. The reason one sometimes uses 'ill-behaved' functions (delta, heaviside, etc) is that they're easier to deal with.