Azure Data Factory SQL Server -> Snowflake Copy Activity

Question

So I've got an ADF pipeline, a copy activity is fed from a for each loop a list of tables to copy into Snowflake. Nearly all tables work perfectly, a few only load the first column.

In Snowflake the command ADF is issuing for the majority of tables is;

COPY INTO SchemaName.TableName
FROM @SchemaName.adfstagename
FORCE = TRUE LOAD_UNCERTAIN_FILES = TRUE
;

On the few that fail to load other columns it is because ADF is specifically choosing to only load the first column with the following;

COPY INTO SchemaName.TableName
(Col1)
FROM (SELECT  $1 FROM @SchemaName.adfstagename)
FORCE = TRUE LOAD_UNCERTAIN_FILES = TRUE
;

I've checked all fields are named the same, data types etc. Does anyone know why ADF would be doing this?

Thanks, Dan

bclark · Accepted Answer · 2022-06-21 22:04:49Z

1

I had the same issue recently and resolved it by unchecking the "First row as header" in the Source dataset (Click the activity -> click Source -> -> click Open next to the Source dataset dropdown). This obviously causes issues if you are trying to retain the column headers for your files but that wasn't an issue for me. Hopefully this resolves your issue as well.

answered Jun 21, 2022 at 22:04

bclark

311 silver badge4 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Dan Over a year ago

Hi, For me this wasn't the solution, after recreating the table (same statement) the copy activity worked.

Collectives™ on Stack Overflow

Azure Data Factory SQL Server -> Snowflake Copy Activity

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related