PIP Constraints Files

From the book "Secret Recipes of the Python Ninja"

Constraints files differ from requirements files in one key way: putting a package in the constraints file does not cause the package to be installed, whereas a requirements file will install all packages listed. Constraints files are simply requirements files that control which version of a package will be installed but provide no control over the actual installation.

Let say you have requirements.txt file with following code

# requirements.txt
pandas

and constraints.txt

# constraints.txt
 # math / science / graph stuff
  bokeh==0.11.1
  numpy==1.10.4
  pandas==0.17.1
  scipy==0.17.0
  openpyxl==2.3.3
  patsy==0.4.1
  matplotlib==1.5.1
  ggplot==0.6.8
  seaborn==0.7.0
  scikit-learn==0.17

executing

pip install -c constraints.txt

will install all packages from requirements.txt and using constraints.txt file for version constraint.

Update: pip 20.3, released Nov 30, 2020, introduced a new resolver that fixes some design issues. It also reimplements the constraints feature. It is now difficult or impossible to use the feature the way I describe below. I do not understand the new constraints implementation, and I no longer use it. I've had success with pip-compile from pip-tools. I specify top-level requirements in requirements.in, and pip-compile generates a requirements.txt with specific versions an package hashes. See requirements.in and requirements.txt in the ichnaea project for a working example.

Original answer below, for pip 20.2 and previous

I think a constraints file is a good way to keep your "true" requirements separate from your full install list.

It's good practice to fully specify package versions in your requirements file. For example, if you are installing django-allauth with Django LTS, pin it to the latest releases (as of my answer):

Django==1.8.12
django-allauth==0.25.2

When you install the package, it ends up installing some required packages as well. So, you add those as well, so that everyone gets the same versions of packages:

Django==1.8.12
django-allauth==0.25.2
oauthlib==1.0.3
python-openid==2.2.5
requests==2.9.1
requests-oauthlib==0.6.1

And then you get the bug report "Doesn't work under Python 3". Oops, python-openid is Python 2 only, and python3-openid is used instead, further requiring defusedxml:

Django==1.8.12
django-allauth==0.25.2
oauthlib==1.0.3
python-openid==2.2.5   ; python_version < '3.0'
python3-openid==3.0.10 ; python_version >= '3.0'
defusedxml==0.4.1      ; python_version >= '3.0'
requests==2.9.1
requests-oauthlib==0.6.1

Now requirements.txt is getting ugly, and it's hard to see the "requirements" of Django and django-allauth in the mess.

Here is a requirements.txt that refers to a constraints file:

-c constraints.txt
Django==1.8.12
django-allauth==0.25.2

And constraints.txt with a helpful comment:

# django-allauth requirements
oauthlib==1.0.3
python-openid==2.2.5
python3-openid==3.0.10
defusedxml==0.4.1
requests==2.9.1
requests-oauthlib==0.6.1

No Python classifiers are needed, because constraints are only installed if a package requires them, and are ignored otherwise. Additionally, if a package stops requiring another package 2 years down the road, fresh installs will stop installing it.

I think this, plus some comments, is a useful way to communicate what packages you are using for the project, and which ones are included because they are dependencies.

I think it gets even more useful if you are using pip 8.x's hash-checking mode, which requires specifying versions for dependencies-of-dependencies. If you go down that path, I recommend hashin to help you manage the hashes. See browsercompat for a complicated requirements setup using constraints and hashes.

Constraint files are very useful when you provide production environments with optimized package compilations for the productive infrastructure.

We provide such environments in docker containers with versions of numpy, scipy, tensorflow, opencv, ... optimized for our production servers. In order to ensure that the requirements files developers set up to make the environment reproducible for them don't overwrite these optimized installations, we use constraint files. They are written as part of the build process. pip is made aware of the constraints by setting the PIP_CONSTRAINT environment variable.

Before, developers always had to make sure their requirements line up with what's available in the prod containers, which can be a pain given that some minor version numbers iterate quite quickly.