Why can I modify a read-only file?

As @Rob already mentioned, you can only do this if you have write access to the directory containing the file. Attempting to do the same thing to a file in, for example, /etc will fail.

As for how vim is doing this, it deletes the file and recreates it. To test this, I created a file owned by root:

echo foo | sudo tee fff

And then proceeded to edit the file with vim in the way that you describe, but attaching the process to strace to see what's happening:

strace vim fff 2> strace.out

I then checked strace.out and found:

unlink("fff")                           = 0
open("fff", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 4
write(4, "foasdasdao\n", 11)            = 11

So, the file was first deleted (unlink("fff")), then a new file of the same name was created (open("fff", O_WRONLY|O_CREAT|O_TRUNC, 0644)) and the modifications I had made were written to it (write(4, "foasdasdao\n", 11)). If you try this at home, you will see that after you edit it with vim, the file will now belong to you and not root.

So, strictly speaking, vim isn't editing a file you have no write access to. It is deleting a file from a directory to which you do have write access and then creating a new file to which, again, you have write access.

As long as you own the parent directory you can remove, or replace a file no matter the permission since you can change the content of the directory :).

Try it with other command like rm, it will prompt you but you can still do it. Make the directory not writable and that should stop it.

Addition:

Just tried it but as long as I own the file I can still modify it, even with the folder read only. However when I change ownership to root:root it cannot open the file for write. So solves the modifying files owned by root (or someone else)

Using w! you are removing the original file (which you are permitted to do) and writing your version instead.

When you have write access to a directory, you can: create, move or delete files within that directory.

$ mkdir foo
$ echo hi > foo/file
$ chmod 777 foo
$ chmod 700 foo/file
$ ls -l foo/file 
-rwx------ 1 ravexina ravexina 7 Aug 31 03:19 foo/file

Now let me switch my user and change the file

$ sudo -u user2 -s
$ vi foo/a # save using w! (I wrote into the file bye)
$ ls -l foo/a
-rwx------ 1 user2 user2 7 Aug 31 03:20 foo/file

Now see what is in there:

$ cat foo/file
bye