Containerize a conda environment in a Singularity container
Using conda-pack it is possible to containerize existing conda environments without re-creating them from environment.yml
. This is particularly useful when the environment doesn't resolve anymore, or when packages have been installed into the environment without conda, e.g. using R
's install.packages
.
pack the environment
conda-pack -n <MY_ENV> -o packed_environment.tar.gz
create this
Singularity
fileBootstrap: docker From: continuumio/miniconda3 %files packed_environment.tar.gz /packed_environment.tar.gz %post tar xvzf /packed_environment.tar.gz -C /opt/conda conda-unpack rm /packed_environment.tar.gz
build the image
singularity build --fakeroot <OUTPUT_CONTAINER.sif> Singularity
For more details, including a Docker/Podman variant, check out my grst/containerize-conda repository.
Limitations: The approach probably only works if the source environment is on a linux x64 machine.
I have found it useful because you can have a container with anaconda3 environment installed and easily create new Environments for your different projects whenever you want.
This is simple and I will go over it step by step:
Create the container in your local machine by the following definition file (you can name it whatever you want. Note that some lines might be avoidable ):
Bootstrap: library From: ubuntu:18.04 Stage: build %post apt-get update && apt-get -y upgrade apt-get -y install \ build-essential \ wget \ bzip2 \ ca-certificates \ libglib2.0-0 \ libxext6 \ libsm6 \ libxrender1 \ git rm -rf /var/lib/apt/lists/* apt-get clean #Installing Anaconda 3 wget -c https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh /bin/bash Anaconda3-2020.02-Linux-x86_64.sh -bfp /usr/local #Conda configuration of channels from .condarc file conda config --file /.condarc --add channels defaults conda config --file /.condarc --add channels conda-forge conda update conda #List installed environments conda list
Then, in order to build the container, run the following commands:
sudo singularity build ContainerName.sif YourDefineFile.def
Now you can create your conda env(you can use common ways or create it by YML file which is an exported file from an existing environment)
For example, I did it with a YML file: First, you need to get into your .sif container as following:
Singularity shell YourContainerName.sif
And then:
conda env create --name envname --file=YourEnvironments.yml
Therefore, After your env is created, you can activate it with the following commands(again first you need to jump into your container):
singularity shell YourContainer.Sif source activate YourEnvName
First, you'll want to get the environment YML for your particular conda environment.
conda activate your_env
conda env export > environment.yml
Here's an example Singularity recipe (in file named 'Singularity' in same directory as 'environment.yml'):
Bootstrap: docker
From: continuumio/miniconda3
%files
environment.yml
%post
/opt/conda/bin/conda env create -f environment.yml
%runscript
exec /opt/conda/envs/$(head -n 1 environment.yml | cut -f 2 -d ' ')/bin/"$@"
Build this with
sudo singularity build conda.sif Singularity
Now, you'll have a functioning container using libraries from your conda environment that can be run anywhere you have Singularity installed!
Example:
singularity run conda.sif ipython
Notes:
Depending on the version of Singularity you're using, you may need to alter $(head -n 1 environment.yml | cut -f 2 -d ' ') with the name of your environment.
Since you can't activate the environment from the runscript, you'll be restricted to binaries installed in your particular environment with the provided runscript.