How does apt upgrade running programs?

In POSIX (and Linux is a POSIX system), at least when using POSIX file systems, files are allowed to remain open after they're deleted (that is, after they're unlinked, by removing the link from a particular file name to the underlying inode object which is what actually identifies a particular string of bytes as the contents of a file). Files in this state continue to be accessible to code that has an open handle to them (a file descriptor in the user land, other structures in the kernel land), although they no longer have a name¹ and anything that tries to open the old name will no longer be able to get them. This means that if you do the following:

$ cat /tmp/foo.txt # step 1
foo
$ rm /tmp/foo.txt # step 2
$ echo "bar" > /tmp/foo.txt # step 3

You're not actually changing the contents of /tmp/foo.txt, rather, you're creating a new, unrelated and independent file that is accessible under the same name whilst the old file no longer is.

Furthermore, POSIX also provides for atomic renames, in which a file is instantly renamed to an existing name, and replaces the original file with no intermediate states. In essence, it's a mechanism which guarantees that nothing will be able to get between step 2 and step 3 above and see the old file deleted but the new one not in place yet.

When apt upgrades a binary that is running, it doesn't actually change the file's contents; instead it uses atomic renames to instantly whisk the new (and completely independent, as described above) file from a temporary name into the old one's place. Anything that had the old file open (this includes processes running it as an executable) will continue to see and use the old contents as if nothing ever happened. But when the process terminates and is restarted, its code will be looked up by name, so the new contents will be used. This also means that it's possible to have processes executing the same named binary, yet running different code.

Note however that the above only applies to individual files. There's nothing to allow atomic replacement of all of an application's resources if they're spread across multiple files, so if, for example, the binary loads a shared library dynamically, or opens some data by name after it's been started, it very well might see something it wasn't expecting and malfunction and/or crash. That's why you get the popup telling you to restart Firefox after it's been upgraded "or you will experience issues".

¹ More properly, they no longer have that particular name, as files (that is, inodes) are allowed to be accessible under multiple names. The vast majority of files however only ever have one name, and when they're unliked from it, they will be deleted² as soon as nothing has them open anymore.

² This deferred deletion is also why, in some cases, you might delete files to reclaim disk space, only to discover that no space was freed. That's because something has those files open and keeps them in that zombie, nameless-yet-alive state.

apt upgrade overwirites files. It doesn't affect running programs, because they are loaded to RAM. If you restart a program, a new version will start.

When upgrading some packages, some triggers or scripts may be run to restart some services if this is needed.

Does it wait until the program terminates?

Maybe, depending on the package.

Does it simply overwrite the files while the program is running? Doesn't this interfere the running program?

It can, depending on the program and the files.

And what about shared libraries?

It depends on whether it has been loaded into memory yet. https://stackoverflow.com/a/7767402/1212596

---

Full answer

Apt (actually, the underlying dpkg) runs scripts when uninstalling and installing a package. If the package includes a long-running program (daemon), it is usually stopped when the old version of the package is removed, and started when the new version of the package is installed.

preinst This script is executed before the package it belongs to is unpacked from its Debian archive (".deb") file. Many 'preinst' scripts stop services for packages which are being upgraded until their installation or upgrade is completed (following the successful execution of the 'postinst' script).

postinst This script typically completes any required configuration of the package foo once foo has been unpacked from its Debian archive (".deb") file. Often, 'postinst' scripts ask users for input, and/or warn them that if they accept default values, they should remember to go back and re-configure that package as needed. Many 'postinst' scripts then execute any commands necessary to start or restart a service once a new package has been installed or upgraded.

prerm This script typically stops any daemons which are associated with a package. It is executed before the removal of files associated with the package.

postrm This script typically modifies links or other files associated with foo, and/or removes files created by the package. (Also see What is a Virtual Package?, Section 7.8.)

https://www.debian.org/doc/manuals/debian-faq/ch-pkg_basics.en.html

It is of course up to the package maintainer to start and stop services in postinst and prerm.