Why can I rename a running executable, but not delete it?

There really is no such thing as renaming a file. A file can have more than one name or no name, so it's not the file that you're renaming but the directory entry. Renaming is an operation on the directory entry, which is not affected by the fact that the file is locked for execution.

It does not allow to delete the executable file and DLLs because Windows maps parts of the executable files into memory as part of the process creation, so it need the file during the lifetime of the process.

Unfortunately I have no true reason why it still allows to rename such files. I guess that this is done to enable the update of the dlls and exe files while they are running to minimize the service interruption time.

The linux (unix in general) in contrast allows to delete an executable file while it is running:

tmp]$ cp /usr/bin/md5sum .;ll md5*; \
(./md5sum /home/pub/iso/FC5/FC-5-i386-DVD.iso & ); \
rm md5sum ; ll md5*;ps -f
-rwxr-xr-x 1 sergey sergey 37276 Oct 16 02:38 md5sum
ls: cannot access md5*: No such file or directory
UID        PID  PPID  C STIME TTY          TIME CMD
sergey    2423  2422  0 02:32 pts/1    00:00:00 -bash
sergey    2533     1  0 02:38 pts/1    00:00:00 ./md5sum /home/pub/iso/FC5/FC-5-
sergey    2536  2423  0 02:38 pts/1    00:00:00 ps -f

I guess it is because a name is just an attribute of the same binary content of the file, so as long as the data is there, the handle, held by the running process for it won't change.