Limiting the total memory usage of PostgreSQL

What do you mean "total" memory? It's the sum of what you allocate to the various pools.

So, in short... don't set the various pools/buffers too big and that's your total memory limit.