Upgrading setuptools on OSX El Capitan

Overview

The problem is conflicting Python libraries, combined with System Integrity Protection (SIP) on Mac OS which protects the system Python libraries.

The best solution in my opinion is to uninstall and re-install your own Python installation and keep that separate to the Python libraries provided by Mac OS which are protected by SIP.

I favour this to disabling SIP as I expect SIP to be a part of any future Mac OS releases and SIP is not the cause here, it just exposes the problem of conflicting Python libraries.

Details

I also had this issue trying to install ansible.

My problem started when I followed the ansible installation instructions for Mac OS, which is to install via pip and to install pip with easy_install as described at Latest Releases Via Pip

Problem is when installing pip this way, the easy_install is the Mac OS provided easy_install at /usr/bin/easy_install but it writes to the easy-install.pth file at /Library/Python/2.7/site-packages/easy-install.pth and that file references the Mac OS provided Python libraries.

Subsequently installing ansible with pip install ansible then reports that the setuptools requirement is satisfied by the Mac OS provided Python library at /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
Ansible requires setuptools but doesn't specify a version. Another dependent package, paramiko, requires setuptools >= 11.3, but this doesn't appear to be checked when installing ansible via pip. When you try to run ansible though it complains that setup tools is 1.1.6 This is the version of setuptools provided with Mac OS, and is now protected by SIP, so it can't be upgraded.

Rather than disable SIP, I fixed this by uninstalling my local Python as hinted at at https://docs.python.org/2.7/using/mac.html#getting-and-installing-macpython and then downloading and installing again.

If you're concerned about the rm, you can mv instead
To uninstall I did

sudo rm -rf /Library/Python
sudo rm -rf /Applications/Python\ 2.7/
sudo rm /usr/local/bin/ansible  # executable
sudo rm /usr/local/bin/python*  # symlinks to /Library/Python/2.7
sudo rm /usr/local/bin/easy_install*
# and so on for references to /Library/Python/2.7 in /usr/local/bin

I then downloaded the 2.7.13 installer package for Mac OS X from https://www.python.org/downloads/ and installed it.

This installed a local python and pip at /Library/Frameworks/Python.framework/Versions/2.7 and symlinks in /usr/local/bin as well as prepending /Library/Frameworks/Python.framework/Versions/2.7 to my $PATH. This keeps everything separate from the Mac OS provided libraries at /System/Library/Frameworks/Python.framework and /usr/bin so that I get

which pip
/Library/Frameworks/Python.framework/Versions/2.7/bin/pip

I then installed ansible with pip install ansible
Then when I run which ansible I get

/Library/Frameworks/Python.framework/Versions/2.7/bin/ansible

and in pip list

setuptools (28.8.0)
six (1.10.0)

and ansible now works for me because the Mac OS Python libraries are not referenced by the new install.

Note that because of the installer setting $PATH, which python is now /Library/Frameworks/Python.framework/Versions/2.7/bin/python and /usr/local/bin/python is a symlink to this.
If you want the system Python you'll have to use /usr/bin/python or change the $PATH

The answer is that you cannot update setuptools on OSX for the factory python that ships with the OS. The reason is that the files in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python cannot be removed or modified by the user. Not only the normal user but even if you have root permissions you will be unable to modify those files.

Sure you can use various other methods to install a newer version of setuptools but this will not override the default system package. Meaning if you go the suggested route of use the --user python flag, or you choose to install them without root into ~/Library, these will not override the system files version.

The reason you or root are unable to modify the system defaults is due to SIP permission restrictions in El Capitan+. You could disable SIP but this is generally not recommended.

Instead the only reasonable solution is to use a python virtualenv.

This happens due to the System Integrity Protection introduced in OS X El Capitan.

Adding --user python to the command allowed this to work.

In other words

pip install --upgrade setuptools --user python