Unable to import lxml etree on aws lambda

Reason:

The .so files compiled on a different OS. (In my case macOS.)

Solution:

The following article was very helpful:

How do I create a Lambda layer using a simulated Lambda environment with Docker?

All I had to do is:

cd /path_to_requirements_txt/

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.7" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.7/site-packages/; exit"

It created python/lib/python3.7/site-packages/ with all the dependencies!

There are modules that cannot be added directly into the site-packages directory to be recognised inside an AWS Lambda environment. When that happens, you have to get an Amazon Linux image from Docker repositories and make your own compiled environment in a container version that will run on AWS Lambda

For example, if you want to use Python 3.6 a good choice will be amazonlinux:2018.03 in case you want to install more packages e.g. pandas, numpy, scipy

    docker run -v $(pwd):/outputs -it amazonlinux:2018.03

Since Amazon Linux is based on Red Hat, you have to install via yum all dependencies when running docker and having already created your virtual environment

    yum update -y
    yum install -y \
      python36 \
      python36-devel \
      python36-virtualenv \
      python34-setuptools \
      gcc \
      gcc-c++ \
      findutils \
      rsync \
      Cython \
      findutils \
      which \
      gzip \
      tar \
      man-pages \
      man \
      wget \
      make \
      zip

For lxml you will also need

    (lambda_docker) bash-4.2# yum install libxml2
    ...
    (lambda_docker) bash-4.2# yum install libxslt
    ...

You install the module as usual

    pip3.6 install lxml

You should see something like

    (lambda_docker) bash-4.2# pip3.6 install lxml
    Collecting lxml
      Downloading https://files.pythonhosted.org/packages/2d/53/34a9f0c79c548e430148837892b6ae91adee571a0e8b6c17bd7ff9c2d12e/lxml-4.3.4-cp36-cp36m-manylinux1_x86_64.whl (5.7MB)
         |################################| 5.7MB 2.0MB/s 
    Installing collected packages: lxml

Then, create your function lambda_function.py and add it in the zipped package pushd-ed and popd-ed from you docker session

    from lxml import etree

    def lambda_handler(event, context):
        print(__name__)
        print(etree.LXML_VERSION)

After created

    zip -g site-packages.zip lambda_function.py

Before uploading, you can check that your zip file contains the lxml directories

    [jonathan@docker lambda_docker]$ unzip -l site-packages.zip 
    Archive:  site-packages.zip
      Length      Date    Time    Name
    ---------  ---------- -----   ----
            0  06-29-2019 23:09   __pycache__/
          277  06-29-2019 23:09   __pycache__/easy_install.cpython-36.pyc
          126  06-29-2019 23:09   easy_install.py
          119  06-29-2019 23:29   lambda_function.py
            0  06-29-2019 23:21   lib/
            0  06-29-2019 23:39   lxml/
            0  06-29-2019 23:37   lxml-4.3.4.dist-info/
            4  06-29-2019 23:37   lxml-4.3.4.dist-info/INSTALLER
         2954  06-29-2019 23:37   lxml-4.3.4.dist-info/METADATA
        13384  06-29-2019 23:37   lxml-4.3.4.dist-info/RECORD
          109  06-29-2019 23:37   lxml-4.3.4.dist-info/WHEEL
            5  06-29-2019 23:37   lxml-4.3.4.dist-info/top_level.txt
         7668  06-29-2019 23:37   lxml/ElementInclude.py
          551  06-29-2019 23:37   lxml/__init__.py
            0  06-29-2019 23:37   lxml/__pycache__/
         3331  06-29-2019 23:37   lxml/__pycache__/ElementInclude.cpython-36.pyc
    ...

Now upload the zip and create an empty test {} in your lambda function

Result

    START RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64 Version: $LATEST
    lambda_function
    (4, 3, 4, 0)
    END RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64
    REPORT RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64  Duration: 0.30 ms   Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 50 MB  

If you prefer an image

Perfectly ready for AWS Lambda

Hope it helps (:

I came across a similar problem and I figured out one quick workaround

Using pre-compiled build of lxml

Download https://github.com/shubh2502/aws-lambda-lxml

1. Folder 3.6.4 and 3.8.0 are lxml versions

2. Inside lxml there are two builds python27 and python36

3. As per AWS Lambda python version choose either one of them

4. Inside python27 and python36 there is lxml folder

5. Zip code with lxml folder and make sure python has the same version

6. In Case of AWS Lambda layer put lxml folder into this structure -

   python/lib/python3.6/site-packages

I spent lots of time in docker and building these stuff, this method was savior for me, I hope this will help you out