AWS Lambda error: not all arguments converted during string formatting

Question

I have a python lambda code which iterates over a loop and starts 17 instances of a reusable glue job by passing run time arguments. I am using boto3 function which is start_job_run() to start the job. Below is the snippet:

    job_keys = {}
    for job_key_name in jobs:
        try:
            job_keys[job_key_name] = glue_client.start_job_run(JobName = glueJobName, Arguments = {'--Path': CONFIG_PATH, '--JobKeyName': job_key_name}) <--- Error is happening here
            logger.info(f'## STARTED GLUE JOB: {glueJobName}-{job_key_name}')
            logger.info(f'## GLUE JOB RUN ID: {job_keys[job_key_name]["JobRunId"]}')
            logger.info(f'## SOURCE: {source}')
        except Exception as e:
            logger.error('ERROR DETAILS', e)
            logger.error(job_key_name)
            pass

The code runs fine and starts 17 executions of the glue job. But there are few intermittent instances when the lambda fails with the error not all arguments converted during string formatting. When this happens the particular job instance does not start.

Note: The max concurrency of the glue job is set to 17 and no. of workers is 2.

i think is coming from logger, please refer this : stackoverflow.com/questions/12843099/… — Abhishek
– Abhishek, Commented Oct 5, 2021 at 6:55
Thank you Abhishek. I will look into it. However, this issue is intermittent and is happening in the middle and not in the first iteration itself. If this was a logging issue, shouldn't it be happening every time?But your point is right, f string is not completely compatible with logging (Ref: stackoverflow.com/questions/54367975/…). Will check.Another point to note is that the iteration where the error occurs that particular job is not executed. Ideally, that job should start atleast as the logger comes after the start_glue_job_run() — Aatisha
– Aatisha, Commented Oct 5, 2021 at 7:33

lynkfox · Accepted Answer · 2021-10-05 12:52:31Z

1

Your error is actually occurring in the Exception itself, but the error is masking another.

logger.error('ERROR DETAILS', e)

when receiving a botocore.ClientError exception will fail at this point with the mentioned error as logger.error has no idea how to process a boto3 ClientError object into a string (there are a lot of additional items on the ClientException object)

If you add the following:

from botocore.exceptions import ClientError
... [ your code ]
except ClientError as e:
    ...[handle the client error specifically]
except Exception as e:
    ...[the rest of your exception handling]

it should allow you to start to see the Client Error issue that is occurring - as it is intermittent, my guess would be that you are running up against glue limits for too many concurrent executions

answered Oct 5, 2021 at 12:52

lynkfox

2,4582 gold badges12 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Aatisha Over a year ago

Thanks for the suggestion. Looks like this is the cause. After handling the client error exception I am now able to see the actual error. By any chance is there a possiblity that the f'string formatting logger.info() could be an issue?

lynkfox Over a year ago

generally no. f'string' formating is generally safe - mostly because if an object can't properly be converted to a string you just end up with [Type: XYZ] instead of an error. Where logger.error('msg', e) tries to pick up some more things from the exception that ClientErrors don't have hooks for. If that helped solve your problem, dont forget to mark it as the answer!

Collectives™ on Stack Overflow

AWS Lambda error: not all arguments converted during string formatting

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related