How do hosting providers prevent the compromise of one website from causing the compromise of another one?

This question is a bit broad, but I think an answer that is a little bit broad will still be helpful. The answer depends on the "kind" of hosting you are talking about. There are three main kinds which I will break down below, but just FYI the names I use are not necessarily industry-standard names. The concepts however are pretty common across the board:

1. Shared hosting (this is the kind of hosting Jeroen talks about in his answer)
2. VPS hosting (aka virtual private servers)
3. Dedicated hosting

Shared Hosting

With a shared hosting setup many websites are running under the same instance of the same OS, and different "websites" are typically separated out by having different user accounts on the same system. cPanel and Plesk are two common software systems that manage such setups (although Plesk can also be used to manage VPS hosting). As a result, you have to concern yourself with securing two primary aspects of your system: making sure that user accounts are properly isolated (see Jeoroen's excellent answer), and making sure that your container manager (Plesk, cPanel, etc...) doesn't have any weaknesses. Those tools do occasionally have vulnerabilities of their own, and if someone manages to break into your container management software, they typically end up with root access to the physical server, or in a worst case scenario, root access to all the servers managed by that system. Also, if you give the users command line access, you have to worry about potential privilege escalation vulnerabilities in the OS itself.

VPS Hosting

With VPS (virtual private server) hosting, the physical server itself is running some sort of virtualization software that then runs any number of "virtual machines" that run a full and separate operating system. Some examples of tools that manage these kinds of setups are the KVM hypervisor or the XEN hypervisor. The advantage of this kind of setup is that it gives the end user (aka the customer of the hosting company) full control over their own system. They can theoretically install any operating system they want, have full root/admin access, and install/manage whatever software they want. The hypervisor keeps each virtual OS separated from the rest and completely sandboxed from one another. Theoretically, it makes it impossible for one compromised host to impact the others at all. In practice of course, things are not always perfect. Although (I believe) it is more rare and harder to exploit, the hypervisors themselves occasionally have their own vulnerabilities that can allow a malicious attacker to take control over the whole system (Spectre and Meltdown were relevant for hypervisors - h/t phyrfox). Like anything else, keeping your systems up-to-date is key.

Dedicated hosting

Some hosting providers offer dedicated hosting, where they basically just manage the hardware for you and provide internet access. They install an OS of your choice on a server and basically just give you root/admin access. Obviously this isn't immediately relevant to your question because this is no longer shared-hosting - no one else to impact. However, they are still inside your network, so proper network access controls are always a must (and this is true for all other hosting instances as well).

Network Management

Edit to add: that last point about network security is worth its own mention. Regardless of whether you use shard hosting, VPS hosting, or dedicated hosting, a hosting company is inherently giving internal network access to an outside party. Without clear access controls, this means that anyone using your hosting services can potentially scan other systems on the network for OS vulnerabilities of their own. For instance, it doesn't matter if your virtualization layer can perfectly isolate the virtual servers running on it if one of the VPS instances can find other instances over the network and gain access via any network-level vulnerabilities in the OS (e.g. heartbleed/eternalblue). Many of the larger hosting companies will allow you (the person using the hosting) to setup a VPC - virtual private cloud - inside their network to isolate your systems from both the internet and other systems on their own network. Presuming that their network rules actually work as promised, this gives the end-user additional ways to protect themselves. Presumably the larger service providers also have active network security that can detect malicious network traffic from inside their own network and shut down accounts appropriately.

In short, the kinds of vulnerabilities you have to worry about depend very much on your hosting configuration. Of course these days there are a whole new slew of hosting options from the large hosting providers (aka serverless infrastructure) which have their own completely separate list of concerns, but I think the above outlines the major concerns for the kinds of hosting you have in mind.

Could one create a vulnerable website on purpose to attack a server of a hosting provider?

Yes, this is definitely possible. Since I have only experienced this with Linux based hosting solutions, I can only share my experience in that scenario.

On several occasions I have managed to compromise an entire shared hosting server which was running DirectAdmin / Plesk.

This was due to insufficient file and folder permissions, any user on the system had read access for other users' home directories (world readable) and thus their domain folders and public_html / private_html sub folders.

If one of the domains on the shared server runs a vulnerable application that, for example, allows users to upload files in a non-secure manner, it is possible to upload a web shell.

Once the attacker has access to the web shell and the file / folder permissions are not properly set, the attacker can browse through the entire file system (as long as folders are world readable of course).

It is recommended to configure the file and folder permission as strictly as possible and contain the vulnerability to a specific domain only.

This is a rather broad question and it assumes that webhosts actually DO suffiently protect you against this problem.

The typical shared web hosting uses:

• Apache mod_suexec
• PHP safe mode, open_basedir and similar measures built in php. This is one of the reasons why almost every shared hosting offer includes php, but rarely perl, python, ruby and so on.
• Possibly running several php-fpm instances using fastcgi. These can even run on another host than the webserver as long as they can read (and write) the files of the corresponding user..
• Restrictive permissions for the user homes.

This still has a lot of room for improvements, which are made by the better offers and neglected by the worse ones.