Unable To Install Cvxpy Into Virtualenv For AWS Lambda

I am trying to run the cvxpy package in an AWS lambda function. This package isn't in the SDK, so I've read that I'll have to compile the dependencies into a zip, and then upload

Solution 1:

For installing cvxpy on windows it requires c++ build tools (please refer: https://buildmedia.readthedocs.org/media/pdf/cvxpy/latest/cvxpy.pdf)

On Windows:

• I created a lambda layer python directory structure python/lib/python3.7/site-packages (refer: https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) and installed my pip packages in that site-packages directory.

pip install cvxpy --target python/lib/python3.7/site-packages

• Then, I zipped the python/lib/python3.7/site-packages as cvxpy_layer.zip and uploaded it to an S3 bucket (layer zipped file max limit is only 50 MB https://docs.aws.amazon.com/lambda/latest/dg/limits.html), to attach it to my lambda layers.
• Now, the layer is ready but the lambda is failing to import the packages as they were installed on a windows machine. (refer: AWS Lambda - unable to import module 'lambda_function')

On Linux:

• I created the same directory structure as earlier python/lib/python3.7/site-packages and installed the cvxpy and zipped it as shown below.
• Later I uploaded the zip file to an S3 bucket and created a new lambda layer.
• Attaching that lambda layer to my lambda function, I colud able to resolve the import issues failing earlier and run the basic cvxpy program on lambda.

mkdir -p alley/python/lib/python3.7/site-packages
pip install cvxpy --target alley/python/lib/python3.7/site-packages
cd alley
zip -rqvT cvxpy_layer.zip .

Lambda layer Image:

Lambda function execution:

Solution 2:

You can wrap all your dependencies along with lambda source into a single zipfile and deploy it. Doing this, you will end up having additional repetitive code in multiple lambda functions. Suppose, if more than one of your lambda functions needs the same package cvxpy, you will have to package it twice for both the functions individually.

Instead a better option would be to try Labmda Layers, where you put all your dependencies into a package and deploy a layer into your Lambda. Then attach that layer to your function to fetch its dependencies from there. The layers can even be versioned. :)

Please refer the below links:

• https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

• https://dev.to/vealkind/getting-started-with-aws-lambda-layers-4ipk