Is there a mechanism that protects applications during library upgrade?

As mentioned by @Kusalananda, usually upgrades are done by removing the old file, and creating a new one with the same name. This will actually create a new file with a new inode, leaving the system free to use the old one as long as it is open.

As a simplified example, stuff like

rm /bin/cat
cp /new/version/of/cat /bin/cat

will create a logically new file, and works even though cat might be running. Same goes for libraries. _{(The above is an example, not a robust way of upgrading a file in the real world.)}

---

Someone could try to change the binary in-place instead of creating a new one with the same name. In this case, at least Linux actually prevents making changes to an executable that is in use:

window 1 # ./cat
window 2 # echo foobar > cat
-bash: cat: Text file busy

However, this doesn't seem to work with dynamically loaded libraries...

I made a copy of libc.so.6 for testing, and filled it with zeroes while it was in use:

window 1 /tmp/lib# LD_LIBRARY_PATH=/tmp/lib ldd ./cat
    linux-vdso.so.1 (0x00007ffcfaf30000)
    libc.so.6 => /tmp/lib/libc.so.6 (0x00007f1145e67000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1146212000)

window 1 /tmp/lib# LD_LIBRARY_PATH=/tmp/lib ./cat
foo
foo

Segmentation fault

(Meanwhile in another window, after the foo, before the segfault)

window 2 /tmp/lib# dd if=/dev/zero of=libc.so.6 bs=1024 count=2000

There's really nothing the program itself could do against this, since I effectively edited its code online.

(This would likely be system dependant, I tested on Debian Jessie 8.5, Linux 3.16.7-ckt25-2+deb8u3. IIRC Windows systems in particular are even more aggressive about preventing in-use files from being modified.)

---

So I guess the answer is that upgrades are usually done in an way that avoids any problems, and this is helped by the filesystem internals. But (on Linux) there don't seem to be any safeguards against actually corrupting dynamic libraries.

Files won't be "properly deleted" if they are unlinked while they are still opened. When they are closed, the disk space that they used will be considered "free" again. This goes for currently running applications and their shared libraries as well.

The only thing that I could see failing would be if a program used dlopen() to load a shared library on demand, or if the program had to access other files on demand such as dictionaries, theme files, or other files that suddenly disappeared.

To illustrate: Running vim in one shell session while deleting the installation of vim in another shell session won't "corrupt" or terminate the currently running vim session. But some things will start to fail, like spell checking for example, which requires vim to open files in its installation.