async http requests in c#

Question

so im making a program which is kind of a web crawler. it downloads the html of a page and parses it for a specific text using regex and then adds it to a list.

to achieve this, i used async http requests. the GET request is sent asynchronously and the parsing operation is performed on the returned html.

my issue, and i'm not sure if it may be simple, is that the program doesn't run smoothly. it will send a bunch of requests, pause for a couple seconds, then increments the items parsed all at once (although the counter is programmed to increment once every time an item is added) so that for example it jumps from 53 to 69 instead of showing, 54,55,56,...

sorry for being a newb but i taught myself all this stuff and some experienced advice would go a long way.

thanks

this is for a specific site where the resulting html is always in the same form with changing variables so regex works fine. — blizz
– blizz, Commented May 17, 2012 at 3:57
but just out of curiosity, is there another method of doing it more efficiently? — blizz
– blizz, Commented May 17, 2012 at 3:58

SLaks · Accepted Answer · 2012-05-17 03:20:45Z

4

That sounds correct.

The slowest part of your task is downloading the pages over the network.

Your program starts downloading a bunch of pages at once, waits for them to arrive, then parses them all almost instantly.

edited May 17, 2012 at 3:20

answered May 17, 2012 at 3:14

SLaks

891k182 gold badges1.9k silver badges2k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

blizz Over a year ago

in that case, can I give priority to the main thread somehow? that is, the thread that is queuing the async requests into ThreadPool? i need this because the main thread is also making a request each time 20 async requests have been made. so whats happening is that its being backlogged behind all the already queued ThreadPool requests and blocking the whole program waiting for its response.

SLaks Over a year ago

@user1115071: Consider using the TPL, which is already optimized for this.

blizz Over a year ago

Please forgive my ignorance as I've never used the TPL. Should I be using it for all threads, or only for the main ones I mentioned?

SLaks Over a year ago

Use Parallel.For* or Task or LINQ AsParallel() and don't use threads or the threadpool directly at all.

Collectives™ on Stack Overflow

async http requests in c#

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related