Does standalone LaTeX exist?

TeXLive has options for running from a USB stick.

Update 2014-02: It has been two years since this response was written. As of February 2014, available packages for USB installation are: TexLive option to run from USB stick (for multiple OSes), MikTex Portable, and TexPortable (both Windows).

Original response:

I guess you're looking for USBTeX: http://www.exomatik.net/U-Latex/USBTeXEnglish

Update 2014-02: USBTeX seems to be no longer maintained; the above URL does not work anymore.

MikTex Portable is exactly what you want. It is a "portable" version of MiKTeX, one of the major TeX distributions, by the MiKTeX project itself. It's the first thing to try before considering repackaged distributions.