Lossy image compression, plus transparency, for the web?

One advantage of PNG is full alpha transparency, which allows you to have smooth edges and shadows in in web designs. The main disadvantage is that it only supports lossless compression, which for complex images means a very large file size.

You are wrong. You can use PNG with lossy compression (8-bit indexed colors) and alpha transparency.

I'm developing pngquant, which generates paletted PNGs with full alpha.

The conversion to palette is pretty close to being lossy compression and gives similarly good results — you can often reduce image sizes by 60-70% with little quality loss.

I've also created Mac GUI for it which includes lossy filter for PNGs (posterization with appropriately selected levels). That technique reduces images by about 30%, but works with 24-bit images.

Are there flash specific layering tricks being used here?

Could be. It's possible to manipulate bitmaps in ActionScript, so you could take the Alpha layer from a simple transparent PNG with no colour data, and combine it with the pixel data from a JPEG.

Are there any formats that allow lossy compression with transparency

Yes, JPEG 2000 and Microsoft's HD Photo. Don't hold your breath for browser support though!