Is a /boot partition always necessary?

To answer the outright question: no, a separate partition for /boot is certainly not necessary in every case.

However, even if you do not split anything else, it is generally recommended to have separate partitions for /, /boot and swap. I would also strongly suggest putting /home on a separate partition as well, even if you do not split the file system hierarchy any further.

The reason for putting /boot on a partition separate from the regular root file system is that you can reduce on-disk file system complexity, which reduces the demands on the boot loader to bootstrap the kernel and initial RAM disk. This becomes particularly interesting if you are running a non-trivial setup - maybe you are running RAID, or an unusual file system such as ZFS on the root partition, or an encrypted root partition. The partition for /boot can then be formatted using e.g. plain ext2fs. Since the partition only needs to be small (a few hundred megabytes is plenty; /boot on my current system is 100 MB, and I do not feel any constraints from that), the downsides of a non-journaled file system such as ext2 need not be that great because checks are so fast anyway, and the relative simplicity of the file system as well as it being largely static might make undetected corruption less likely. That the boot loader doesn't need to natively understand an exotic file system or disk setup is another possible advantage.

The reason for putting /home on a separate partition is that, even if something goes rogue and starts filling it up (or if you do so yourself), you never risk it causing problems for non-user processes (which rarely run out of /home) or the boot process itself. Also, it becomes much easier to reinstall the OS, or switch distributions, if you can simply tell the installer to leave /home as it is and reformat / to fill it with its own files. As a last-ditch recovery effort if something truly goes wrong software-wise, this can make a major difference.

Running a swap file is not a recommended setup on Linux, and I'm not even sure most common distributions' stock kernels support (or allow) file-based swap. There are multiple reasons for this, one large reason being performance (largely due to the risk for fragmentation). Swap is already hideously slow compared to RAM (it's more of a stopgap measure than a fix), so there is little reason to make it even slower by risking fragmentation. And a swap partition can safely be shared between Linux distributions in a multi-boot environment; perhaps not a consideration in the general case, but certainly a consideration for some. A swap partition can also be placed on a separate physical disk with different characteristics; maybe a SSD these days, or a 10krpm drive running off a separate controller channel (that last is less a consideration nowadays when every SATA disk is on a separate channel, but could make a difference back in the PATA days).

Personally these days, I separate /, /boot, and swap, with / on my current system being 100 GB (I'm nowhere near using all of that; curent usage is 8.9 GB, and that gives me everything I need in terms of software and then some). The rest of the primary disk is made up of a single file system, which by personal convention I mount at /da (disk a). A second disk would be /db, then /dc and so on. (I doubt the FHS really approves of this scheme, but it works well for me in practice.) I then bind-mount /home into /da, meaning I can move the physical directory around without having to worry about updating every single path referencing it or repartitioning just because I realized that I am running a bit short of disk space for /home.

I don't think it is mandatory anymore. In the past, it was necessary because of some limitations with the BIOS. It was impossible to have a bootable partition after 8 or 64 Go (I don't remember exactly). Then, if you wanted to have a dual boot, you could put a first small partition (/boot), and the bootable windows partition, and after, the other Linux partitions. There is a long time, there was no /boot directory at all. Kernel was directly on /.

The difference is filesystem optimization. Generally each partition have it's own tables for files, descriptors etc. You can install Linux on a single '/' partition without '/boot', '/swap', /usr' etc, but this single partition internal tables will soon grow huge and some operations will become a bit slow. So different partitions are used to provide separated, optimized containers for different kind of tasks.