How to accelerate a nested loop Jupyter Notebook Python with big tables

Question

this is my code to iterate two tables simultanous

for i in df_MailBox_SentService.index :
    val_s = df_MailBox_SentService.at[i,'MailId']
    for j in df_TBL_SentMail.index :
        if val_s == df_TBL_SentMail.at[j,'ID']:
            df_MailBox_SentService.at[i,'UnifiedMailNumber']=df_TBL_SentMail.at[j,'MailNbr']

you are assigning values to a position determined exclusively by i df_MailBox_SentService.at[i,'UnifiedMailNumber'] but then you have an inner loop that will keep overwriting on that position every time the conditional is triggered (since j is not used to determine the position to write to). The loops don't make much sense to me, unless the conditional is guaranteed to trigger only once, in that case you should break the inner loop to improve performance — user20396240
– user20396240, Commented Feb 1, 2023 at 15:15
It looks like you are using pandas. In pandas you can join two dataframes, so you don't need to loop over them with for loops. If you need help with that, provide a full code example - including code to create the dataframes - so you can get help. Also useful, say how big the big tables are. — 576i
– 576i, Commented Feb 1, 2023 at 15:21

jjislam · Accepted Answer · 2023-02-01 15:34:16Z

1

Try this:

mask = df_MailBox_SentService['MailId'] == df_TBL_SentMail['ID']
df_MailBox_SentService.loc[mask, 'UnifiedMailNumber'] = df_TBL_SentMail.loc[mask, 'MailNbr']

answered Feb 1, 2023 at 15:34

jjislam

5613 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Aymen Ragguem Over a year ago

this is the error : Can only compare identically-labeled Series objects

Collectives™ on Stack Overflow

How to accelerate a nested loop Jupyter Notebook Python with big tables

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related