Is there any penalty to wearing stolen items?

No. But if you commit a crime and the guards catch you, they will keep all your stolen goods.

No, they won't. But watch what you take; if you take highly valuable items or many moderately valuable items then you may find yourself being attacked by hired mercenaries with a contract to kill you. You will find the reason the mercenaries were hired and the name of who hired them in the letter. Also, getting caught doing any crime will result in the items you have stolen being taken back.