Is it possible to limit the memory an application is allowed to use?

You can use Microsoft's Application Verifier. It doesn't give a lot of options for memory management, but allows to run apps in 'resource constrained' environment. It's primary usage is for profiling/debugging, but it can be used just to constrain bad behaving apps.

Here's a cure that might be worse than the disease - run the application in a virtual machine. The virtual machine will have a hard limit on the RAM it can utilize.

There's obviously some overhead in creating the virtual machine, but for Firefox you could probably run it in a very small Linux distribution.

I know this isn't a general approach to limiting the amount of RAM available to the applications, but here is a nice guide to limiting the amount of RAM available to firefox.

I'm not sure if there is a general approach to limiting memory usage for single applications (except for maybe java applications).