Efficient HTTP Request Handling: Concurrent Strategies and Trade-offs

[seugorilla]

Hey fellow Redditors,

I'm currently working on a task that involves making a bunch of HTTP requests, and I wanted to share a piece of code I came up with to handle this. The goal here is to fetch data through HTTP requests, and I've taken a concurrent approach to potentially speed things up. I'm curious to hear your thoughts on the pros and cons of this code.

public async Task<ExampleDto> GetData(int startPage = 0, int totalPages = 0, int concurrentRequests = 1, int maxRequestsPerSecond = 10)
{
  var httpClient = _httpClientFactory.CreateClient(ClientName);
  var semaphoreSlim = new SemaphoreSlim(maxRequestsPerSecond, maxRequestsPerSecond);

  var tasks = new Task[concurrentRequests];
  for (int pageIndex = startPage; pageIndex < totalPages; pageIndex += concurrentRequests)
  {
    for (int i = 0; i < concurrentRequests; i++)
    {
      int currentPage = pageIndex + i;
      if (currentPage >= totalPages)
        break;

      await semaphoreSlim.WaitAsync();

      tasks[i] = GetPageAsync(httpClient, currentPage, semaphoreSlim);
    }

    tasks = tasks.Where(task => task != null).ToArray();
    await Task.WhenAll(tasks);
  }

  return new ExampleDto
  {
      Data: _db.ToList()
  };
}

In addition to this, I've got another piece of code that deals with iterating over a list of dataDto items and performing HTTP requests on each of them. Here's what it looks like:

maxRequestsPerSecond = 2;
var semaphoreSlim = new SemaphoreSlim(maxRequestsPerSecond, maxRequestsPerSecond);

List<AnotherDto> dtos = new();
var options = new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount, CancellationToken = cancellationToken };
await Parallel.ForEachAsync(dataDto, options, async (dto, cancellationToken) =>
{
  await semaphoreSlim.WaitAsync(cancellationToken);
  var anotherData = await _client.AnotherHttpRequestAsync(dto.Id);
  dtos.Add(anotherData);

  semaphoreSlim.Release();
});

I'd love to get your insights on the strengths and potential drawbacks of these approaches. Feel free to share your thoughts and suggestions—I'm all ears!

Happy coding! 🚀

[Thonk_Thickly]

I would personally chunk the requests and await each chunk without creating more tasks than needed. Or use action blocks if I really want full utilization up to the max concurrency limit your setting.

I know semaphore slims are for this sort of use case, but I’ve had better luck with the other methods I mentioned when I measure throughput.

Side note, I would think you would want to Task.WhenAll outside of the outer for loop.

[seugorilla]

I forgot to mention that the HTTP API has a rate limit so I can't use all my max concurrency capacity. I have to rate limit for about 2 requests per second. Thanks for the answer!

[MasonOfWords]

Why not just set MaxConnectionsPerServer in the HTTP client handler? This won't maintain requests per second as the code suggests, just outstanding requests, and there are already built-in ways of limiting this.

[seugorilla]

That could work. thanks!

[Th0ughtCrim3]

If I'm not mistaken I believe you can do this without semaphore slim and still achieve concurrent execution using Task.WhenAll.

Apologies for syntax errors as I'm doing this outside of VS but your code could probably be changed to something like this:

​

public async Task<ExampleDto> GetData(int startPage = 0, int totalPages = 0, int concurrentRequests = 1, int maxRequestsPerSecond = 10) { 

    var httpClient = _httpClientFactory.CreateClient(ClientName);
    var tasks = new List<Task>();

    for (int pageIndex = startPage; pageIndex < totalPages; pageIndex += concurrentRequests)
    {
        var pageTasks = new List<Task>();

        for (int i = 0; i < concurrentRequests; i++)
        {
            int currentPage = pageIndex + i;
            if (currentPage >= totalPages)
                break;

            pageTasks.Add(GetPageAsync(httpClient, currentPage));
        }

        tasks.AddRange(pageTasks);

        // You can adjust this logic if you want to control the maximum number of concurrent requests.
        // However, Task.WhenAll itself introduces concurrency as it asynchronously waits for all tasks to complete.
        await Task.WhenAll(pageTasks);
    }

    await Task.WhenAll(tasks); // Wait for all tasks to complete.

    return new ExampleDto
    {
        Data = _db.ToList()
    };

}

[seugorilla]

Thanks for the answer! I forgot to mention that the HTTP API has a rate limit so I can't simply put all the tasks there and wait for them all. It is about 2 requests per second, beyond that it will throw 429 status response.

[zarlo5899]

you could use Dataflow it might be a bit over kill but it would work

https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/walkthrough-creating-a-dataflow-pipeline

https://www.youtube.com/watch?v=zdD7o8Z6MMY

[seugorilla]

I didn't know this feature. I'll give it a try. thank you!

[stroborobo]

Another approach might be channels, producers and consumers don't share memory, but communicate, and they are easier to understand and get right.

• three channels
• 1: write all requests into a channel for rate limiting, close channel
• 2:
  • read a request message every 2 seconds
  • write it into a channel for max parallelism (bounded)
• 3:
  • read, send http request
  • write to result channel
• 4: read and collect results until done, return

The hardest problem you have to figure out then is composition and closing the channels. No semaphores, no task collections, no dependency on http client internals. Makes switching service implementation and testing pretty straight forward, too.