How to Use Filter Activity Output as a Source in Copy Activity in Azure Data Factory Pipeline

Question

I'm fairly new to Azure Data Factory and need help with a pipeline I'm building. My goal is to read data from a CSV file stored in an Amazon S3 bucket, filter out records where the Status column is 'active', and then copy the filtered data into a SQL Server sink. I’m using a Lookup activity to fetch the data and passing its output (@activity('Lookup1').output.value) to a Filter activity. The filter condition I’ve set is @equals(toLower(item().Status), 'active'). This gives me the filtered records, but I’m unsure how to pass this filtered output into the Copy activity so that only the active records are written to the SQL Server. Any guidance on how to achieve this would be greatly appreciated. PS : I want to avoid use of Dataflows due to some other reasons.

I tried the following approach : The output of Filter activity cannot be directly used in Copy Activity, since Copy Activity requires a Source dataset and what I get from filter activity is an array of data, so I am not able to use this as a source. So we tried using Foreach activity and placing Copy Activity inside it, but encountered same issue.

Hijesh V L · Accepted Answer · 2025-06-05 13:41:23Z

0

The easiest solution would be to load the file to a staging table as it is and then create a 2nd copy activity to filter the active records from the staging table. In the 2nd copy activity source side you can use an sql query for filtering.

Another approach is using dataflows where you can filter the records based on the active column value before loading to the table

answered Jun 5 at 13:41

Hijesh V L

2792 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Tarun Sahu Jun 6 at 6:58

Thankyou, I actually don't think it is possible using the Filter so I tweaked the solution you gave and used a single copy activity followed by a stored procedure which filtered the data in the sink table.

Collectives™ on Stack Overflow

How to Use Filter Activity Output as a Source in Copy Activity in Azure Data Factory Pipeline

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related