How do I quickly switch between weapons and spells?

That is the favoriting feature, on the PC to add an item or spell to your favorites you have to press F. Then you can access all your favorites quickly by pressing Q, a list of favorites will pop up in the lower left corner and the game will be paused.

On the PC you can also bind items and spells to the number keys.

You can set either weapons or spells or whathaveyou as Favorites in your inventory (keyboard default is F), and then, during combat, you can change them quickly by pressing the Favorites button (keyboard default is Q), which brings up the Favorites menu. From there, you can select which weapon or spell you want to use in whichever hand you want to use them in.

On the PS3 you have the favorites you access by pressing up or down on the direction pad. The left and right on the direction pad are Quick Keys. I usually have the left set to my Daedric bow and the right set to my Daedric warhammer. To make something a favorite go into your inventory, select your favorite thing and press triangle. If you want to assign the favorite to the quick keys, select it from the favorites list and press left or right on the direction pad.