When a process forks is its virtual or resident memory copied?

In modern systems none of the memory is actually copied just because a fork system call is used. It is all marked read only in the page table such that on first attempt to write a trap into kernel code will happen. Only once the first process attempt to write will the copying happen.

This is known as copy-on-write.

However it may be necessary to keep track of committed address space as well. If no memory or swap is available at the time the kernel has to copy a page, it has to kill some process to free memory. This is not always desirable, so it is possible to keep track of how much memory the kernel has committed to.

If the kernel would commit to more than the available memory + swap, it can give an error code on attempt to call fork. If enough is available the kernel will commit to the full virtual size of the parent for both processes after the fork.

Don't worry, it makes a lazy copy (copy-on-write). The virtual memory addresses of both processes point to the same pages initially, but when the forked process tries to modify it, it actually makes a physical copy of the page (from then on, that page resides in two places in your RAM).

Beware, none of the reported memory footprints actually tell you how much of RAM the process is using. Because of swapping, memory sharing and other issues with virtual memory, it's impossible to know for sure. Some parts of the memory space are shared libraries (where to count them?), some refer to non-RAM memory (other hardware devices), some are currently swapped-out, some are not copied yet (copy-on-write) and so on. Read this:

https://lwn.net/Articles/642202/

There is kernel setting

/proc/sys/vm/overcommit_memory

Citation from excellent article:

Since 2.5.30 the values are: 0 (default): as before: guess about how much  
overcommitment is reasonable, 1: never refuse any malloc(), 2: be precise 
about the overcommit - never commit a virtual address space larger than swap 
space plus a fraction overcommit_ratio of the physical memory. Here 
/proc/sys/vm/overcommit_ratio (by default 50) is another user-settable 
parameter. It is possible to set overcommit_ratio to values larger than 100. 
(See also Documentation/vm/overcommit-accounting.)

This applies to forks as well as regular malloc. I.e. if you set it to 0, fork will be copy on write. Copy on write means that once app forked, it's both copies will share memory pages util child or original starts changing memory.

In most distributions I know overcommit is 0. But if you set it to 2, all memory pages will be fully backed by real memory and in some cases under high memory pressure will be more stable, but some programs (I faced gitk) which rely on overcommits will fail.