Is it possible for a vulnerability in one application to be exploited to attack another application on the same server?

If the Magento vulnerability led to a shell, then that shell could be used to get root access by a privilege escalation vulnerability. When that happens, the attacker has complete control over the system.

The short answer is a qualified "Yes." Especially if the software systems are running in the same file system space and memory space. If an attacker gets root, the system is pwned. Having said that, there are a couple of configurations that would make the pwnage much more difficult and the HIDS should have time to raise alarms about the activity.

The most common is to have each application run in its own chroot jail. It should be possible to lock things down tightly enough with something like SELinux or RBAC on Windows or Solaris. They are not airtight, the intruder would be making so much noise trying to figure out how they are configured and how to grant him/herself privileges necessary to continue the attack, again, by this time the HIDS would have given the game away.

An even safer configuration is to have each application run on its own virtual machine. If the virtual machines are running on top of an operating system, the attacker would have to break completely out of the virtual machine on which the vulnerable application was running and break into the other one. Assuming that the virtual machines and the host machine are each running its own HIDS, the alarm(s) will go off way before the attack on the second machine is completed. If they are running on bare metal, I think it's safe to say that whomever would attack that system does not have the resources to complete the attack.

IMHO, having each application run in it's own virtual machine is a good idea anyway. It should do the job and the implementation is far and away the easiest to implement and manage . . .