ADF upsert error locating key column in the interim table

Question

I have a pipeline with copy data task where I upsert some data from On-Prem SQL into Synapse. The setup is such that I first establish the list of tables and then, in ForEach, I dynamically set Key Columns (from a variable) and do an upsert. The strange part is that occasionally copy data task fails with this error

ErrorCode=SqlUpsertKeyColumnsInvalid,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Sql upsert key column '[some-column-name]' does not exist in the table '[some-schema-name].[InterimTable_9688bbfd-bb01-4a10-b6f6-9832bcb12512]'.,Source=Microsoft.DataTransfer.ClientLibrary,'

When I re-run the pipeline, the error goes away for one table but may occur for another table. There is about 100 tables involved in this copy operation and the amount of data being upserted is not substantial (few hundred records if not less).

Thank you in advance!

Kabo · Accepted Answer · 2024-12-19 09:21:16Z

0

If you do it by looping with parallel executions, the variable might be overwritten by other parallel executions. To avoid this, you need to deactivate parallel executions or execute another pipeline in the loop. The secondary pipeline needs the parameters used in the loop and won't have them overwritten by parallel loops.

answered Dec 19, 2024 at 9:21

Kabo

1

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

ADF upsert error locating key column in the interim table

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related