Containerize a conda environment in a Singularity container

Using conda-pack it is possible to containerize existing conda environments without re-creating them from environment.yml. This is particularly useful when the environment doesn't resolve anymore, or when packages have been installed into the environment without conda, e.g. using R's install.packages.

  1. pack the environment

    conda-pack -n <MY_ENV> -o packed_environment.tar.gz
    
  2. create this Singularity file

    
    Bootstrap: docker
    
    From: continuumio/miniconda3
    
    %files
        packed_environment.tar.gz /packed_environment.tar.gz
    
    %post
        tar xvzf /packed_environment.tar.gz -C /opt/conda
        conda-unpack
        rm /packed_environment.tar.gz
    
  3. build the image

    singularity build --fakeroot <OUTPUT_CONTAINER.sif> Singularity
    

For more details, including a Docker/Podman variant, check out my grst/containerize-conda repository.

Limitations: The approach probably only works if the source environment is on a linux x64 machine.


I have found it useful because you can have a container with anaconda3 environment installed and easily create new Environments for your different projects whenever you want.

This is simple and I will go over it step by step:

  1. Create the container in your local machine by the following definition file (you can name it whatever you want. Note that some lines might be avoidable ):

    Bootstrap: library
    From: ubuntu:18.04
    Stage: build
    
    %post
    
    apt-get update && apt-get -y upgrade
    apt-get -y install \
    build-essential \
    wget \
    bzip2 \
    ca-certificates \
    libglib2.0-0 \
    libxext6 \
    libsm6 \
    libxrender1 \
    git
    rm -rf /var/lib/apt/lists/*
    apt-get clean
    #Installing Anaconda 3 
    wget -c https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
    /bin/bash Anaconda3-2020.02-Linux-x86_64.sh -bfp /usr/local
    #Conda configuration of channels from .condarc file
    conda config --file /.condarc --add channels defaults
    conda config --file /.condarc --add channels conda-forge
    conda update conda
    #List installed environments
    conda list
    
  2. Then, in order to build the container, run the following commands:

    sudo singularity build ContainerName.sif YourDefineFile.def
    
  3. Now you can create your conda env(you can use common ways or create it by YML file which is an exported file from an existing environment)

    For example, I did it with a YML file: First, you need to get into your .sif container as following:

     Singularity shell YourContainerName.sif 
    

    And then:

     conda env create --name envname --file=YourEnvironments.yml
    
  4. Therefore, After your env is created, you can activate it with the following commands(again first you need to jump into your container):

    singularity shell YourContainer.Sif
    
    source activate YourEnvName
    

First, you'll want to get the environment YML for your particular conda environment.

conda activate your_env
conda env export > environment.yml

Here's an example Singularity recipe (in file named 'Singularity' in same directory as 'environment.yml'):

Bootstrap: docker

From: continuumio/miniconda3

%files
    environment.yml

%post
    /opt/conda/bin/conda env create -f environment.yml

%runscript
    exec /opt/conda/envs/$(head -n 1 environment.yml | cut -f 2 -d ' ')/bin/"$@"

Build this with

sudo singularity build conda.sif Singularity

Now, you'll have a functioning container using libraries from your conda environment that can be run anywhere you have Singularity installed!

Example:

singularity run conda.sif ipython

Notes:

Depending on the version of Singularity you're using, you may need to alter $(head -n 1 environment.yml | cut -f 2 -d ' ') with the name of your environment.

Since you can't activate the environment from the runscript, you'll be restricted to binaries installed in your particular environment with the provided runscript.