installing python packages without internet and using source code as .tar.gz and .whl

This is how I handle this case:

On the machine where I have access to Internet:

mkdir keystone-deps
pip download python-keystoneclient -d "/home/aviuser/keystone-deps"
tar cvfz keystone-deps.tgz keystone-deps

Then move the tar file to the destination machine that does not have Internet access and perform the following:

tar xvfz keystone-deps.tgz
cd keystone-deps
pip install python_keystoneclient-2.3.1-py2.py3-none-any.whl -f ./ --no-index

You may need to add --no-deps to the command as follows:

pip install python_keystoneclient-2.3.1-py2.py3-none-any.whl -f ./ --no-index --no-deps

If you want to install a bunch of dependencies from, say a requirements.txt, you would do:

mkdir dependencies
pip download -r requirements.txt -d "./dependencies"
tar cvfz dependencies.tar.gz dependencies

And, once you transfer the dependencies.tar.gz to the machine which does not have internet you would do:

tar zxvf dependencies.tar.gz
cd dependencies
pip install * -f ./ --no-index

We have a similar situation at work, where the production machines have no access to the Internet; therefore everything has to be managed offline and off-host.

Here is what I tried with varied amounts of success:

1. basket which is a small utility that you run on your internet-connected host. Instead of trying to install a package, it will instead download it, and everything else it requires to be installed into a directory. You then move this directory onto your target machine. Pros: very easy and simple to use, no server headaches; no ports to configure. Cons: there aren't any real showstoppers, but the biggest one is that it doesn't respect any version pinning you may have; it will always download the latest version of a package.

2. Run a local pypi server. Used pypiserver and devpi. pypiserver is super simple to install and setup; devpi takes a bit more finagling. They both do the same thing - act as a proxy/cache for the real pypi and as a local pypi server for any home-grown packages. localshop is a new one that wasn't around when I was looking, it also has the same idea. So how it works is your internet-restricted machine will connect to these servers, they are then connected to the Internet so that they can cache and proxy the actual repository.

The problem with the second approach is that although you get maximum compatibility and access to the entire repository of Python packages, you still need to make sure any/all dependencies are installed on your target machines (for example, any headers for database drivers and a build toolchain). Further, these solutions do not cater for non-pypi repositories (for example, packages that are hosted on github).

We got very far with the second option though, so I would definitely recommend it.

Eventually, getting tired of having to deal with compatibility issues and libraries, we migrated the entire circus of servers to commercially supported docker containers.

This means that we ship everything pre-configured, nothing actually needs to be installed on the production machines and it has been the most headache-free solution for us.

We replaced the pypi repositories with a local docker image server.