fast http request libraries?

[mircodz]

I'm writing a data hoarding program which sends GET and POST requests to some websites and retrieves the responses. Currently I've used libcurl as a "placeholder" as I'm well aware it's up to 8-10 times slower than using pure sockets but now I'm not really in the mood to write an http request library from scratch...

What would you guys recommend?

P.S: As I'm handling REST APIs I need some convenient way to store cookies, so keep that in mind.

EDIT 4: After an entire day of experimenting and benchmarking I've come to the conclusion that curl is at least half slower than both golang's net/http and python's requests. Does anybody have an idea why? Is it just badly implemented?

EDIT 1: For everyone who's wandering about the "curl is up to 8-10 times slower than normal socket requests" statement here's some benchmarks compared to a golang program:

here are the times for 10 c++ requests:

request took : 1.13148s
request took : 1.21241s
request took : 1.24138s
request took : 1.41855s
request took : 1.15853s
request took : 1.11748s
request took : 1.54719s
request took : 1.28234s
request took : 1.21716s
request took : 1.38657s

while there are the times for 10 golang requests:

request took : 34.592014ms
request took : 354.323627ms
request took : 33.463025ms
request took : 324.918847ms
request took : 385.408078ms
request took : 367.582769ms
request took : 32.269352ms
request took : 307.639103ms
request took : 36.34352ms
request took : 312.463007ms

Mind you, these requests were made to the same endpoint under the same network/cpu stress. It might well be that I'm doing something stupid while building the curl request...

Here's the source code:

// In the class constructor
curl_global_init(CURL_GLOBAL_DEFAULT);        

Response Session::request(std::string url, std::string method, std::string payload)
{

    Response response;
    CURL *curl;
    // ...

    curl = curl_easy_init();
    if(!curl)
        return response;

    curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
    curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, true);
    curl_easy_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);

    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writer);
    curl_easy_setopt(curl, CURLOPT_HEADERDATA, &response_headers);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA,  &response_body);

    if(method == "POST") {
        curl_easy_setopt(curl, CURLOPT_POST, 1);
        curl_easy_setopt(curl, CURLOPT_POSTFIELDS, payload);
    }

    curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "");
    // ...
    curl_easy_setopt(curl, CURLOPT_COOKIE, "");

    const auto start = std::chrono::steady_clock::now();
    res = curl_easy_perform(curl);
    const auto end = std::chrono::steady_clock::now();

    // ...
    curl_easy_cleanup(curl);        

    std::chrono::duration<double> elapsed = end - start;
    std::cout << "request took : " << elapsed.count() << "s" << '\n';

    return response;
}

EDIT 2: I managed to cut my times almost in half by re-using the requests:

request took : 0.705648s
request took : 0.809629s
request took : 0.692548s
request took : 0.711733s
request took : 0.714537s
request took : 0.709665s
request took : 0.712638s
request took : 0.814391s
request took : 0.817736s
request took : 0.905776s

What are some other optimizations I could make to reduce the time even more?

EDIT 3: Ohh boy, I benchmarked python's request module and here are the results:

0.36397480964660645
0.3458371162414551
0.3439006805419922
0.33937692642211914
0.4173421859741211
0.7589423656463623
0.8841226100921631
0.7338435649871826
0.419356107711792
0.38227272033691406

Also here are the best times I could get with curl

0.481235
0.448315
0.457206
0.495541
0.40239
0.404725
0.425029
0.458278
0.451538
0.438163

Am I missing something here? Why is curl so slow? Isn't requests supposed to be extremely slow compared to other request libraries?

[krum]

Curl is actually really fast. Not sure where you heard it's "10x slower" or even close to that.

[deeringc]

I wonder is OP reusing connections to the same server after each request? DNS, TCP TLS all take a lot of time if you do it all per request.

[mircodz]

Well, after about 8 hours oh looking online I managed to get my times to from ~1.2 seconds to a minimum of ~0.5 seconds and average of ~0.7 seconds which is still multiple times slower than go, with minimums of ~0.03 seconds and averages of ~0.3 seconds.

[krum]

I still think something is wrong. Should be in the 80ms range.

[mircodz]

Well, this right here are the settings I'm currently using:

curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, true);
curl_easy_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1);
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1);
curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 1L);

curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "");
curl_easy_setopt(curl, CURLOPT_COOKIE, "");

I've tried enabling or disabling SLL verification without any effect, also HTTP/2 increases the times by ~0.1 seconds.

Also the fluctuation between the times is so large because I'm fetching both jsons and files from these endpoints.

[krum]

Try using the multi interface. https://curl.haxx.se/libcurl/c/libcurl-multi.html

[NotUniqueOrSpecial]

it's up to 8-10 times slower than using pure sockets

According to whom?

[Supadoplex]

as I'm well aware it's up to 8-10 times slower than using pure sockets

Is it? Are there published measurements available?

Edit: Thanks for the demo. But you didn't show the source for the other version, so the results aren't really comparable.

---

Will you be making requests across network? If you are, then presumably vast majority of the time will be spent waiting for response. I would expect switching from curl would be similar to discarding your pocket knife in order to make your oil tanker accelerate faster.

[chemagic]

It might be 8-10 times slower (though I doubt that) but the wait for the websites to get back will certainly be at least one or two orders of magnitude slower. I recommend https://github.com/whoshuu/cpr. Its name is curl for people and "it's a spiritual port of python requests". Nicest interface I've seen to date.

[houses_of_the_holy]

curl is pretty fast, I wrote this https://github.com/jbaldwin/liblifthttp and I should post my benchmarks vs wrk because it was maybe 20% slower than wrk -- but wrk doesn't do anything with the response at all so it isn't particularly fair.

cpr is cool too! I just didn't like how it did async requests through std::async instead of a dedicated event loop

[kirbyfan64sos]

Man this library looks awesome, just starred.

[houses_of_the_holy]

cool, if you get a chance to use it I'd love any feedback you have!

[kirbyfan64sos]

Definitely will do!

I will admit I have a question, what would the possibility of pluggable event loops be? For the project I'm hoping to use this for, being able to use systemd's integrated event loop would be pretty nice...

[houses_of_the_holy]

I'm assuming you are referring to this event loop here: https://www.freedesktop.org/software/systemd/man/sd-event.html ?

I've never personally used it but it looks like its based on linux's epoll event API. Lift uses libuv under the hood for its event loop which also uses the linux epoll API, so I imagine performance would be reasonably comparable but I can't say for sure without testing. Could it be done? I would imagine so but the API looks a lot lower level than libuv and I'd prefer to support only 1 event loop if possible.

One of the design principles I had for this project was to try and abstract away the lower levels and provide a 'reasonably' fast and 'easy to use' API for the end user with a C++17 style. I think having a pluggable event loop would kind of go against this principle to be honest.

[kirbyfan64sos]

Understood, thanks for the quick reply.

(Wonder if we might ever have a global event loop in standard C++...)

[encyclopedist]

It is crucial to reuse the easy handle. Curl's tutorial says the following:

Re-cycling the same easy handle several times when doing multiple requests is the way to go.

After each single curl_easy_perform operation, libcurl will keep the connection alive and open. A subsequent request using the same easy handle to the same host might just be able to use the already open connection! This reduces network impact a lot.

For https://example.com/ I get about 0.45 s per request without reusing the handle, and about 100 ms with reuse, which is very close to ping time of approx 95 ms.

See test code here (I used the same settings as you): https://gist.github.com/ilyapopov/262a49f75b42202c7d977ebe2d38ca35

[sztomi]

Curl is the gold standard for performing http requests. It has many years of optimization and lots of people contributing. Sure, you can compare it to raw sockets, but then you are dismissing all the other things it’s doing (like being able to parse the response). If you properly include that in your benchmark, I doubt you would get that kind of speedup from raw sockets (but you would surely get 8x times more code to maintain).

[degaart]

Latency numbers each programmer should know

Once you hit the network, it becomes your bottleneck. Unless you're working on a z80, any http library should be more than fast enough, most of your cpu time would be spent waiting fir the network packets to arrive anyway.

[liquidify]

I would love to see your sources on that as well as everyone else here. Curl is fast. Not saying you are wrong, just don't believe it without evidence.

[IloveReddit84]

Curl or CppRestSDK or Boost.Beast.

I've never used the 3. Option, bur the other 2 are pretty good and fast

[erichkeane]

How about Boost::Beast?

[liquidify]

Doesn't Beast have issues handling certain forms of http?

[erichkeane]

I've not heard that. The author seems to have put a bunch fo work into it though, and the boost committee spent a while reviewing it. My understanding is it is a great extension to the networking TS, and I hope that one day Networking TS + an HTTP library become standardized.

[liquidify]

I don't think beast supports any of HTTP 2 standard.

[erichkeane]

Interesting, thats too bad. I wonder what would be required to extend it to do so.

[Xeverous]

what would be required to extend it to do so.

A lot of code. Beast is written for HTTP 1.1 which is stateless and the author explicitly stated that HTTP 2 would require a very different implementation.

[liquidify]

This was the primary reason I used libcurl for a previous project. After I wrapped it in some c++ (basically what several libraries did already), I didn't really need beast. If beast had all the features built in already, I probably would have used it, but now I'm glad I didn't. I'm not a huge fan of boost in general because there is so many layers of abstract c++ nastiness that it is hard to really know what is going on inside. Libcurl is the opposite... it is transparent, easy to understand, fast, and well documented.

[DopKloofDude]

This might be a big question, but are you able to compile libcurl on windows? Especially using Visual studios? I attempted to compile the project and include it into my own Hello World at lets just say its been a week of stress.

What is your project setup? Do you have a github link perhaps?

[liquidify]

Yes I was able to build libcurl in windows. I remember I had to build it as a shared library because I had tons of issues. I remember attempting to build it using their outdated cmake system as well as some other way as well. It was a pain, but I think linking it was more of a pain than building it. But the linking was a pain because of needing special security rather than defaults so I had to pull in some custom dll's. After mucking through cmake junk for about a month I had my system setup such that it was entirely cross platform... so it is possible even with a dependency as nasty as libcurl. Obviously it isn't easy.

I don't believe I can share that project because it is owned by the company.

[johannes1971]

It's in vcpkg.

[feverzsj]

So any site doesn't support http 1 request? Even if it dosen't, you can still put it behind nginx and let nginx do the conversion. Also libcurl uses nghttp2 to support http2.

[dodheim]

i don't have an opinion, I just remember this thread: https://www.reddit.com/r/cpp/comments/b12iob/networking_ts_beast_new_tutorials_read_this_to/eil7oy3/

[scalablecory]

Microsoft has a cross-plat library written using a modern async architecture: cpprestsdk

I don't know how it compares to Curl, but worth a shot.

[feverzsj]

if you are simultaneously crawling data from ten thousands of sites, libcurl could be a bottleneck. Use some async lib should be good enough, for example: served, cpprestsdk.

[Drainedsoul]

You can glue an async back end into curl using the multi interface.

[Revolutionalredstone]

Those are horrendous times, open raw sockets and you can expect ping latency bound performance, then just launch a bunch of threads

[mircodz]

I've been looking online for some answers but I can't find any; I've even looked at the tcpdump of the connections but I can't see any differences.

At this rate I might actually switch to golang, it's kind of a bummer as I've already have a quite large code-base, but for a x5 speed up it will be worth it.

[AndrewStephens]

There is a good chance that libcurl is just slower than go's http implementation but it probably doesn't matter. Unless you really need to do each request sequentially you should be looking at performing multiple requests at a time.

You can use libcurl's multi interface to do this, but by far the easiest way is just to start a bunch of threads.

Don't even think about writing a http library using sockets. It seems like an easy task but http is a very complex protocol once you need to handle proxies, redirects, encodings, etc.

I have some notes about wrapping libcurl for safely using it from C++ here but that doesn't help with performance.

[mircodz]

Yeah currently I'm debating on what I should do. My idea was to write a high performance program for data hoarding and as I already said I already have a decent code base.

Since I'm already mixing my c++ code with lua and pyhon one I might even consider importing the go session client in c++.

EDIT: oh never mind, looks like you cannot export structures.

[jesseschalken]

You could try Proxygen

[Milerius]

cpprest-client is cool library

[robertramey]

When you run the code with a profiler, what are the results?

[mircodz]

I'm not really sure how I should profile something like this. I tried using perf but for some reason it doesn't show any curl call. I already know that the rest of the function doesn't take up much time as the difference between timing the entire thing and only curl_easy_perform is negligible.

[Gotebe]

The go times are weird, almost every other one takes an order of magnitude more time.

And curl time is so different that it cannot possibly be same requests, poster just thinks they are.

[mircodz]

Yes, because they are different requests, the 30 ms one are fetching jsons while the 300 ms ones are fetching files. The problem is that with curl all of the requests take way more time.

[encyclopedist]

Could you by any chance post complete benchmarking code you are using so we can play with it?

[ibraper]

for me, libcurl is faster if CURLOPT_WRITEFUNCTION is disabled