Out of swap - what happens?

It depends on the settings you're running with, in particular memory overcommit (/proc/sys/vm/overcommit_memory; see man 5 proc for details).

If memory overcommit is disabled, the editor's (and possibly other programs attempting at the same time) attempt to allocate memory will fail. They'll get a failure result from the system call. Its up to each program to handle this, though an unfortunately common result is for the program to crash. The editor may also, for example, just refuse to open the file.

If memory overcommit is enabled, then the system call requesting memory may well succeed. In that case, when the memory is actually accessed, the kernel will notice its out of memory, and kill a process to reclaim memory. That process may or may not be the editor. The choice is governed by the oom_score (the result of several kernel heuristics) and oom_score_adj (configured) of each process on the system. Those are also in that proc(5) manpage.

There is a huge problem in Linux in this case if you approach to the out-of-memory condition - you will notice that whole your system becomes totally unresponsive because it starts a lot of swapping. Even your mouse cursor may become so 'slow' that you can't start a terminal and kill an offending memory eater process manually. This is because of huge number of disk operations.

To avoid this situation, I personally usually completely disable swap, so the Linux kernel is always responsive and in worst case the out of memory (OOM) killer will kill some process. The logic of which process is killed by OOM depends on the kernel version.

So answer is no - don't enable dynamic swap allocation. You will face machine hangs.

It's easy to try it out with program that just constantly allocates some memory in a loop. Save this program to a text file, memeater.c:

#include <stdlib.h>

int main() {
    for (;;) {char* mem=malloc(4096); mem[0]=1;};
}

Then compile it:

$ gcc memeater.c -o memeater

and run:

$ ./memeater

Try it with swap, without swap, and with your dynamic swap allocation.

Also, keep in mind that in most cases this OOM condition happens because a bug in software (memory leak) or you did something wrong like 'load this 10 GB file in editor' or 'run too many graphical file resizes in parallel' and do the conclusion: Do you need swap or not?