How to limit application memory usage?

Yes, VSZ is virtual memory. As to RLIMIT_AS, where did you find the paragraph quoted above? Since setrlimit(2) is a Linux system call, I do not see how it could possibly monitor malloc(3), a library function. Instead, it can only work with brk(2), sbrk(2), and mmap(2) -- this is also what its manpage (checked of Scientific Linux) suggests. However, the total amount of memory requested via these functions is virtual memory, so RLIMIT_AS indeed limits virtual memory. (This is, again, in accordance with the setrlimit(2) manpage.)

Unfortunately, you cannot limit RSS under Linux (this would be ulimit -m). You can try ulimit -d (RLIMIT_DATA), but this will include mmap(2) only since Linux 4.7, typically used for large allocations. Another possibility would be to limit virtual memory, but with such a large difference between RSS and VSZ, this might be difficult.

Many processes share some of its memory with other processes, e.g. libc is used by nearly every process but only mapped in memory once, but it counts towards the virtual memory usage of every process. Limiting memory usage that's only used by a certain process (mostly RSS) can be done using cgroups. See answers to How to limit the total resources (memory) of a process and its children for how to do it. This will limit the total memory of a process and its children.