How can I simulate a modem over a VOIP connection?

To save your sanity, don't.

Modems (and fax) over VoIP are extremely sensitive to timing, and usually break over VoIP, it might even seem to work at first, but it's certain to break.

Adding virtual machines just makes the problem worse.

There are "short haul modems" around which can emulate a phone line.

Another option is just to attach the two modems back to back:

http://www.jagshouse.com/modem.html

I have a similar need, and I've been searching for hours and hours for a solution. The closest I've found is a free but not very well-written program called "TimeFax" (http://sourceforge.net/projects/timefax/), which installs a virtual COM port and connects it to an H.323 receiver. You can make Windows install a legacy 14k modem on that COM port, and theoretically the program will take the COM port data and transform it to H.323 and pipe it to whichever receiver you specify. Then all you need is a VoIP H.323 termination, and you might be good.

Note that to get TimeFax working on Windows Vista or 7, you have to disable desktop composition in the compatibility settings. You'll also have to manually download the 2 dll files that it is missing, which it will complain about when you first try to run it. A quick Google search finds them easily though.

You'll still have issues with VoIP compression, but if you lower the speed, you might be able to achieve something that works. A few years ago I routed a dial-up modem through a Vonage connection, and it worked surprisingly well if I kept it at slow speeds (like 14k), so I know that dial-up over VoIP is possible, I just haven't quite got a completely software solution working.