Using async and collections

Question

I wish to download a large number of files with httpclient, performs some time consuming but not expensive computation on them, and then add the result to my database after running some query that shows that it is not already there.

How can I do this conceptually (just the locations of the awaits and the like would be helpful)

I currently have the following:

get list of addresses add (await the web page download, then continue processing) to a list of Task foreach element of the list, await on it, and then add it to the database.

However, it seems that this is essentially running it serially.

How should this be designed?

i3arnon · Accepted Answer · 2014-03-31 12:05:32Z

3

I would set up a pipeline using TPL Dataflow. You post the addresses and the actors are:

Web page download
Processing
Add to DB

Use async wherever you can (as long as the operation is truly asynchronous) and set a high MaxDegreeOfParallelism to allow TPL to choose the optimal value by itself.

answered Mar 31, 2014 at 12:05

i3arnon

117k33 gold badges334 silver badges358 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

soandos Over a year ago

Do you mean use the primitives that it provides (actionblock, bufferblock, etc) or something else? If you mean the primitives, can you please explain which ones? I am not at all familiar with TPL.

i3arnon Over a year ago

Well, yes. The easiest way is to use ActionBlock for all of them. You only need to give it a method (WebPageDownload for example) and remember to "send" it to the next ActionBlock at the end. The more "correct" way is to also use TransformBlocks

soandos Over a year ago

Is there a nice way of having a trasformblock later in the chain post to a transform block that is earlier in the chain?

i3arnon Over a year ago

@soandos There's no technical problem with it. It just needs to have a reference to the earlier block. But if you do that it stops being a simple pipeline and you're in danger of a continuous loop.

soandos Over a year ago

But what am I looping on? is download.InputCount != 0 && Download.OutputCount != 0 && process.OutputCount != 0 && process.InputCount != 0 (the looping ones) correct?

|

Todd Menier · Accepted Answer · 2014-03-31 12:15:46Z

1

I would get the downloads/processing running in parallel and await them all to complete. The code would look something like this:

// get a collection of "hot" Tasks running in parallel
var tasks = myCollection.Select(x => DownloadAndProcessAsync(x));

// await the completion of all Tasks
await Task.WhenAll(tasks);

edited Mar 31, 2014 at 12:15

answered Mar 31, 2014 at 12:05

Todd Menier

39.7k18 gold badges156 silver badges185 bronze badges

2 Comments

i3arnon Over a year ago

There's no point of waiting for all of the tasks to complete before moving on with what's already finished

Todd Menier Over a year ago

The method inside the lambda was intended to download and perform all processing for a single item. Updated the method name to reflect that.

Collectives™ on Stack Overflow

Using async and collections

2 Answers 2

6 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related