How do I know when to upgrade RAM in Vista?

When you can afford it, buy it. You always need more memory, especially in Vista.

I recommend using the Reliability and Performance Monitoring Tool to see if your system is memory starved, specifically the memory section. Look for hard faults/sec:

A hard fault (also known as a page fault) occurs when the page of the referenced address is no longer in physical memory and has been swapped out or is available from a backing file on disk. It is not an error. However, a high number of hard faults may explain the slow response time of an application if it must continually read data back from disk rather than from physical memory.

Try using resource monitor and looking at the percentage of physical memory used.