What's using my swap (Ubuntu)?

The swap usage patterns you describe don't sound surprising. They're consistent with some permanently-running processes having rarely-used pages. During the day, due to the high activity, the rarely-used pages are almost always in the swap. At night, there's more room for them in RAM.

You can get a glimpse of how much memory of various kinds each process is using in top or htop. Neither show swap usage by default, but both can be configured to (top: press f and switch on the SWAP column; htop: press F2, add the NSWAP column). You can get more information about a particular process with cat /proc/12345/vmstat where 12345 is the process ID. Note that “how much swap a program is using” is not completely well-defined, as some pages are shared by several processes.

There are two major kinds of competitors for RAM: process memory (which can be swapped out) and disk caches (which can be re-read from a file). There is no reason to always prioritize process memory over disk caches: it's better to swap out a rarely-used portion of process memory than keep reading a file into memory again and again. The figures you give, with about half (say 30%–70%) of the memory devoted to disk caches, is typical for systems that have a reasonable amount of RAM for the tasks they're supposed to do.