Where do confiscated items go?

When they're confiscated, they're placed in an evidence chest. These chests can be found in the jails themselves and in guard houses.

If you have the chutzpah, you can re-enter the jail once you serve your time and seek it out. It's usually around the same area as the belongings chest: the location is different depending on the jail, but checking the barracks or the entrances (or even sometimes right next to your cell) are generally a good bet.

It goes into the evidence chest of the jail/dungeon where you where arrested.