Use AWS Lambda to run Sagemaker Batch Transform job

Question

I would like to place a csv file in an S3 bucket and get predictions from a Sagemaker model using batch transform job automatically. I would like to do that by using s3 event notification (upon csv upload) to trigger a Lambda function which would do a batch transform job. The lambda function I have written so far is this:


import boto3
sagemaker = boto3.client('sagemaker')

input_data_path = 's3://yeex/upload/examples.csv'.format(default_bucket, 's3://yeex/upload/', 'examples.csv')
output_data_path = 's3://nooz/download/'.format(default_bucket, 's3://nooz/download')

transform_job = sagemaker.transformer.Transformer(
    model_name = y_xgboost_21,
    instance_count = 1,
    instance_type = 'ml.m5.large',
    strategy = 'SingleRecord',
    assemble_with = 'Line',
    output_path = output_data_path,
    base_transform_job_name='y-test-batch',
    sagemaker_session=sagemaker.Session(),
    accept = 'text/csv')

transform_job.transform(data = input_data_path, 
                        content_type = 'text/csv', 
                        split_type = 'Line')

The error it returns is that object sagemaker does not have module transform What is the syntax I should use in Lambda function?

The error is telling you that transformer is not a valid method that you can call on a sagemaker client. You can review the available methods here boto3.amazonaws.com/v1/documentation/api/latest/reference/… — Paolo
– Paolo, Commented Mar 27, 2022 at 17:44

dingus · Accepted Answer · 2022-03-31 03:54:30Z

While Boto3 (boto3.client("sagemaker")) is the general-purpose AWS SDK for Python across different services, examples that you might see referencing classes like Estimator, Transformer, Predictor and etc are referring to the SageMaker Python SDK (import sagemaker).

In general I'd say (almost?) anything that can be done in one can also be done in the other as they use the same underlying service APIs - but the purpose of the SM Python SDK is to provide higher-level abstractions and useful utilities: For example transparently zipping and uploading a source_dir to S3 to deliver "script mode" training.

As far as I'm aware, the SageMaker Python SDK is still not pre-installed in AWS Lambda Python runtimes by default: But it is an open-source and pip-installable package.

So you have 2 choices here:

Continue using boto3 and create your transform job via the low-level create_transform_job API
Install sagemaker in your Python Lambda bundle (Tools like AWS SAM or CDK might make this process easier) and instead import sagemaker so you can use the Transformer and other high-level Python APIs.

Collectives™ on Stack Overflow

Use AWS Lambda to run Sagemaker Batch Transform job

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related