Synchronous Concurrent HttpClient Usage

Question

I am trying to use HttpClient synchronously but when I make many concurrent requests it stops working. I wrote two tests, one for asynchronous usage and one for synchronous usage. TestMethod always returns response after 4 seconds. Asynchronous test works fine. Almost all requests are timed out in synchronous test, only ~20 last requests are successful. I tried both using a single HttpClient for requests and creating new HttpClient instance for each new requests. No difference. Maybe it has something to do with this deadlock.

I am using VS2013 with .NET Framework 4.5.1 targeting Framework 4.5. I get HttpClient via NuGet: <package id="Microsoft.Net.Http" version="2.2.15" targetFramework="net45" />

I don't want to use HttpClient asynchronously just yet because it means that I have to rewrite my whole app. Any ideas what am I doing wrong here?

// all 800 requests complete successfully
[Test]
public async void Asyncmethod()
{
    var sw = new Stopwatch();
    sw.Start();

    var client = new HttpClient();
    client.Timeout = TimeSpan.FromSeconds(15);

    var tasks = Enumerable.Range(0, 800).Select(x => Task.Run(async () =>
    {
        try
        {
            var swx = new Stopwatch();
            swx.Start();
            var response = await client.GetStringAsync("http://localhost:7002/TestMethod").ConfigureAwait(false);
            swx.Stop();
            Console.WriteLine(x + " " + response + " in " + swx.ElapsedMilliseconds + " ms.");
        }
        catch (Exception e)
        {
            Console.WriteLine(x + " Exception: " + e.Message);
        }
    })).ToArray();

    await Task.WhenAll(tasks);

    sw.Stop();
    Console.WriteLine(sw.ElapsedMilliseconds);
}

// almost all of 800 requests time out
[Test]
public void Syncmethod()
{
    var sw = new Stopwatch();
    sw.Start();

    var client = new HttpClient();

    var tasks = Enumerable.Range(0, 800).Select(x => Task.Run(() =>
    {
        try
        {
            var swx = new Stopwatch();
            swx.Start();
            var response = client.GetStringAsync("http://localhost:7002/TestMethod");
            if (response.Wait(15000))
            {
                swx.Stop();
                Console.WriteLine(x + " " + response.Result + " in " + swx.ElapsedMilliseconds + " ms.");
            }
            else
            {
                swx.Stop();
                Console.WriteLine(x + " timed out.");
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(x + " Exception: " + e.Message);
        }
    })).ToArray();

    foreach (var task in tasks)
        task.Wait(60000);

    sw.Stop();
    Console.WriteLine(sw.ElapsedMilliseconds);
}

Stephen Cleary · Accepted Answer · 2013-10-22 12:53:49Z

8

I do recommend that you use await for HttpClient instead of Wait. If you truly want synchronous requests, consider using WebClient which supports synchronous methods.

That said, I believe the reason you're seeing that behavior is due to ServicePointManager.DefaultConnectionLimit, which will throttle the number of requests to a given server. If you want to have large numbers of concurrent requests to the same server (synchronous or asynchronous), then you'll need to increase that limit (it is 2 by default for UI applications).

answered Oct 22, 2013 at 12:53

Stephen Cleary

460k78 gold badges721 silver badges858 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Paulo Morgado Over a year ago

await - after all the question is tagged C#-5.0

Thresh Over a year ago

The problem is, if there is no timeout, none of the requests is completed. I get around 20 completed requests only after 780 are canceled due to timeout.

Thresh Over a year ago

Using await in not an option, I'll have to rewrite everything else too. Yes, WebClient works perfectly. But I still don't understand why HttpClient hangs.

Panagiotis Kanavos Over a year ago

The DefaultConnectionLimit is a VERY good reason to hang. The default is 2 which means that getting even 20 completed requests is not bad. Just increase the limit

Community · Accepted Answer · 2017-05-23 11:46:37Z

2

Yes, It's seems like deadlock.

This code works perfect

public static void Syncmethod()
    {
        var sw = new Stopwatch();
        sw.Start();

        var client = new HttpClient();

        var tasks = Enumerable.Range(0, 800).Select(x => Task.Run(() =>
        {

            var swx = new Stopwatch();
            swx.Start();
            var result =
            client.GetStringAsync("http://yandex.ru").ContinueWith(task =>
            {
                try
                {
                    swx.Stop();
                    Console.WriteLine(x + " " + task.Result + " in " + swx.ElapsedMilliseconds + " ms.");
                    return task.Result;
                }
                catch (Exception e)
                {
                    swx.Stop();
                    Console.WriteLine(x + " Exception: " + e.Message);
                    throw e;
                }
            }, TaskContinuationOptions.AttachedToParent);

        })).ToArray();

        foreach (var task in tasks)
            task.Wait(60000);

        sw.Stop();
        Console.WriteLine(sw.ElapsedMilliseconds);
    }

I guess you need additional logic to handle timeout. May be one more task that completes in 15 sec, for example.

Look to this post HttpClient.GetAsync(...) never returns when using await/async

And as described above you may hang your async example by simple change

var response = client.GetStringAsync("http://yandex.ru").GetAwaiter().GetResult();

When you write async code never ever use block execution and await result.

edited May 23, 2017 at 11:46

CommunityBot

11 silver badge

answered Oct 22, 2013 at 8:46

sh1ng

2,9934 gold badges26 silver badges39 bronze badges

2 Comments

Thresh Over a year ago

First of all, in your example the task will be completed before the request is completed if request is long enough. Second, how would I write a function that returns something from response using ContinueWith?

sh1ng Over a year ago

Updated. But you have to rewrite you code to use Task<string> without waiting.

Collectives™ on Stack Overflow

Synchronous Concurrent HttpClient Usage

2 Answers 2

4 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related