1

I was able to successfully follow this tutorial here to zip a scikit-learn package and create a layer. Now I have two layers loaded, one for scikit-learn and one for numpy and scipy that AWS already has. You can see that below

enter image description here

When I try to run my lambda now, I get an error that states that

"errorMessage": "Unable to import module 'lambda_function': No module named 'pandas'"

Thus I tried to do the same process with zipping up a pandas file together but when I try to load it, I exceed that 50mb maximum. Is there anyway of loading in pandas and sklearn together so that I do not go above this maximum limit?

Improve this question
2
  • 3
    Upload your zip first to s3, and then link the S3 zip. Commented Jun 8, 2021 at 0:23
  • @Marcin How would that help? You still have to create the layer either from a zip file or an S3 bucket. Commented Jun 8, 2021 at 16:03

3 Answers 3

Reset to default
1

You can use docker container as Lambda images. This will allow you to use way bigger libraries.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

The 50 MB limit is for upload only. From docs:

50 MB (zipped, for direct upload)

If you want to have greater layers, you have to upload them to S3 first. Then the 50 MB limit does not apply and you are constrained by:

250 MB (unzipped, including layers)

Improve this answer

3 Comments

Okay, I uploaded the zip files to s3, but to add a layer you still have to create them in Lambda and add them. So I am not sure how that will help.
@justanewb Your issue is that you exceed 50 MB limit from what I understand. Now it will not apply.
I apologize, I must not understand. Could you explain in steps through AWS how I achieve what you are saying. So far I have all my zipped python modules in a S3 bucket.
1

You can use the following repo which includes a process for creating lambda layers. It includes configuration for sklearn (with pandas, numpy and scipy).

You clone it and use a CLI with details like python version and architecture.

https://github.com/imperva/aws-lambda-layer

Note: Docker must be installed on the machine you run the process (you can use docker desktop) since it uses docker build

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.