Proper way to parse requirements file after pip upgrade to pip 10.x.x?
First, I believe parsing requirements.txt
from within setup.py
is not a good idea. It should be the other way around, install_requires
in setup.py
or setup.cfg
should be considered as some kind of source of truth, and files such as requirements.txt
should be generated from there. But everyone has different needs, that lead to different workflows.
So with that said...
It is possible to parse a relatively simple requirements.txt
file from a setuptools setup.py
script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources
.
It could more or less look like this:
#!/usr/bin/env python
import pathlib
import pkg_resources
import setuptools
with pathlib.Path('requirements.txt').open() as requirements_txt:
install_requires = [
str(requirement)
for requirement
in pkg_resources.parse_requirements(requirements_txt)
]
setuptools.setup(
install_requires=install_requires,
)
Again, this will work only for simple requirements.txt
files. See Requirements parsing in the documentation page for pkg_resources
to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not supported and it will cause a failure.
Word of caution
As stated already, this is not recommended. The requirements.txt
file and the list of "install dependencies" are two different concepts, they are not interchangeable.
But if you insist on writing a setup.py
install script that reads a requirements.txt
file then make sure that this requirements.txt
file is included in the "source distribution" (sdist) otherwise installation will fail, for obvious reasons.
Since setuptools version 62.6 it is possible to write something like this in setup.cfg
:
[options]
install_requires = file: requirements.txt
Alternatively in pyproject.toml
:
[project]
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
dependencies = requirements.txt
Same words of caution as above apply:
- only very simple files are supported
- the file must be added to the sdist
Additionally, it is considered a "beta" feature for now.
Notes:
- See also this other answer: https://stackoverflow.com/a/59971469/11138259
- https://caremad.io/posts/2013/07/setup-vs-requirement/
- https://setuptools.pypa.io/en/latest/history.html#v62-6-0
The solution of Scrotch only works until pip 19.0.3
, in the pip >= 20
versions the PipSession module was refactored. Here is a solution for the imports that works for all pip
versions:
try:
# pip >=20
from pip._internal.network.session import PipSession
from pip._internal.req import parse_requirements
except ImportError:
try:
# 10.0.0 <= pip <= 19.3.1
from pip._internal.download import PipSession
from pip._internal.req import parse_requirements
except ImportError:
# pip <= 9.0.3
from pip.download import PipSession
from pip.req import parse_requirements
EDIT: modified to support pip>= 19.0.3
I don't agree with the accepted answer. The setup.py
file can get ugly real fast if you have a large project with a lot of dependencies. It is always good practice to keep your requirements in a separate .txt
file. I would do something like this -
try:
# pip >=20
from pip._internal.network.session import PipSession
from pip._internal.req import parse_requirements
except ImportError:
try:
# 10.0.0 <= pip <= 19.3.1
from pip._internal.download import PipSession
from pip._internal.req import parse_requirements
except ImportError:
# pip <= 9.0.3
from pip.download import PipSession
from pip.req import parse_requirements
requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())
if __name__ == '__main__':
setup(
...
install_requires=[str(requirement.requirement) for requirement in requirements],
...
)
Throw in all your requirements in requirements.txt
under project root directory.