Replaced third party code with git submodules, now I can't switch branches

Unfortunately, I think this is just one of the drawbacks of using submodules. These problems are described in a section called "Issues with Submodules" in Pro Git, but in short, the simplest workaround is to move the submodule directory out of the way before switching to the master branch:

mv Vendor Vendor.moved
git checkout master

Similarly, when you change to develop, you should do:

git checkout develop
mv Vendor.moved Vendor