How do I set my default Music folder?

As you mentioned, when you modify (rename / move) Music folder it will point to the new folder.
To force it to point to a different folder, modify ~/.config/user-dirs.dirs file (it's a text file) and set the XDG_MUSIC_DIR parameter to point to your Music.

Or use ubuntu tweak

You download it from here : http://ubuntu-tweak.com/