Why does this command create an enormously large file?

You cannot tell cat to use multiple standard out that way, the last redirection takes precedence so:

cat file1.txt >> file2.txt >> file1.txt

is equivalent to:

>> file2.txt ; cat file1.txt >> file1.txt

which obviously quickly fills the file system, given the fact the source file being the destination too grows indefinitely provided file1.txt is large enough not to be read at once.

Most modern cat implementations should detect the recursivity and abort:

Solaris cat:

cat: input/output files 'file1.txt' identical

Gnu cat:

cat: file1.txt: input file is output file

They can be fooled anyway with something like:

cat < file1.txt | cat | cat  >> file2.txt >> file1.txt

A nice not so useless use of cats ...

I wasn't able to reproduce this in a Bash shell:

# non-empty file1
$ echo 1 > file1.txt

$ cat file1.txt >> file2.txt >> file1.txt 
cat: file1.txt: input file is output file

1 file gets created with 0 length but then I get the above message:

$ ls -l
total 4
-rw-rw-r-- 1 saml saml   2 Sep 10 19:35 file1.txt
-rw-rw-r-- 1 saml saml   0 Sep 10 19:35 file2.txt

Based on @jlliagre's answer I'm not sure why I'm getting the 2 files. It may be dependent on the cat implementation.

EDIT #1

@jlliagre updated his answer to show this code which he states as equivalent:

>> file2.txt ; cat file1.txt >> file1.txt

So now I know why I'm getting the empty file2.txt. This notation is legal:

>> file2.txt

And will create an empty file.