Is there a zonal statistics tool to calculate median and mode raster values for each polygon?

I haven't calculated mode values before (most frequently occurring value), but at least for quantile values you could try to use the plugin I coded (LecoS). Although not primarily intended for this use it is able to calculate some "zonal statistics" for polygon overlays.
How to:

• Install numpy and scipy for your System and QGIS. On Windows choose the OSGEO4W advanced Install and select python-numpy and python-scipy. On Linux consult your linux package manager/compiler.
• Download the experimental(!) LecoS plugin via the plugin downloader. The rest should be pretty self-explaining.

Please pay attention that the plugin is still experimental and calculation might fail sometimes! If you want a clear procedure suited for your needs than i would advise you to look into numpy and also R with the raster and sp packages. I am also happy to see detailed bug reports and i will do the best i can to fix them. However if the calculation fails for you, you better lookout for other tools as well.

The v.rast.stats module in GRASS GIS is very powerful and will allow you to calculate univariate statistics (n, min, max, range, mean, stddev, variance, coeff_var, sum) from a raster map based on polygon boundaries.