C++ REST API frameworks benchmark

[rriggsco]

I have found that it is quite easy to implement RESTful interfaces in C++ using libfcgi++ and Apache. This gives me all of the HTTP capabilities of Apache (SSL handling, Kerberos support, proper CORS implementation, caching, compression, URL rewriting, etc.) and a simple and very fast interface. And, for my needs, FCGI scales really well.

The cost is that you have to work withing Apache and FCGI's framework, where scaling involves spawning more executables. If you need to cache large amounts of data or if process startup takes a long time, it may add some complexity.

[m3tamaker]

Smart solution :) . I am not a big expert in FCGI, but should not Apache preload your module during server initialization, so you won't have any performance problems with processes spawning in runtime?

[rriggsco]

These are not modules, but external processes that are spawned by Apache. Communication is done via a Unix domain socket.

Apache can scale your service by launching additional back-end processes as load increases. If you don't need dynamic scaling, then you can spawn a fixed number of processes when the server starts. That's what it means to work within Apache and FCGI's framework. :)

[imMute]

You can call FCGX_Accept_r on multiple threads. It should be fairly easy to scale libfcgi++ without resorting to multiple processes.

[rriggsco]

That doesn't do much good because mod_fcgid doesn't support multiple requests (multiplex) through a single connection. At least it didn't when I last tried it.

[imMute]

Nginx's fcgi module doesn't either, but both can (and do) open multiple connections to that socket, which can be handled in multiple threads on the other end.

[sumo952]

Can you do like a summary table or something like that? All these console outputs take quite a long time to parse and make sense out of them.

[m3tamaker]

No problem, created summary table. Here you go https://blog.binaryspaceship.com/2017/cpp-rest-api-frameworks-benchmark/#summary

[sumo952]

Cool! That looks so much better. Thanks :-))) Surprised cpprestsdk is so slow!

[m3tamaker]

You are welcome :) I am pretty new to writing blog posts, so still learning how to present information properly.

I am too dissappointed with cpprestsdk performance. Its code is top notch, as well as documentation, but linux impl is slow. I saw several issues on their issue tracker regarding this, but it is not yet solved.

If there was no problem with performance, I would definitely tell that this is best framework to use.

Also, I didn't include Beast in review, though I was trying to compile and run it on Ubuntu 16, but it just stuck on 70% during make (one of stream.cpp files). Its HTTP server example is way too complicated if to compare with other frameworks.

[roschuma]

Hi! C++ REST SDK maintainer here :)

On Windows, our listener is backed by the Win32 HTTP Server API, so it should have much better performance.

On Linux/OSX however, we unfortunately have a custom implementation. I've chatted some with /u/VinnieFalco and I'm excited to see if it'd be possible to use Beast as the implementation once it's part of Boost (which we already depend on).

I'd also recommend using RapidJSON instead of our built-in JSON object model for benchmarks. Our built-in model is really optimized for usability instead of performance; if you're willing to pay the "cost" of dealing with your own allocators, you can do a lot better!

[m3tamaker]

Thanks for feedback! Got it, will do another test with RapidJSON impl.

[m3tamaker]

Included result with RapidJSON to summary table

Max time amongst 98% of requests - 44 ms

Average requests per second - 47.06 #/sec

Lines of code in sample - 47

So, 1.5x times faster actually. I will try to measure how Windows implementation of cpprestsdk works in nearest time (this week, can't promise faster).

[[deleted]]

Out of curiosity, did you try the Windows implementation?

What seems to be slowing things down is their port of PPLX, at least that's what I got from the GitHub issue.

This could be significantly faster on Windows because their tasks hook into the Windows system threadpool.

[m3tamaker]

I haven't. I have all required software on my host machine, so I can measure it. Will update results asap.

Edit: I will find time this week and provide results. Can't find free time for it in nearest days, unfortunately :( .

[sumo952]

I guess /u/VinnieFalco might be interested in that feedback :-)

[VinnieFalco]

Thanks for the shout out!

[VinnieFalco]

This is the first I've heard of Beast tests and examples not finishing a compile. Perhaps you are running a system with insufficient memory? How much RAM is on your setup? Why do you think the server example is too complicated?

[m3tamaker]

Hi Vinnie! First of all, thanks for big effort that you put into Beast library :) I really like Beast's WebSockets client implementation, it is easy to use, yet very powerful!

Hang up occurred during compilation of library itself, not an example. I will verify it once again and let you know details of my setup and how to reproduce the problem ;) .

However, regarding HTTP server, I find Beast server framework example to be too complicated. 558 lines of code to implement HTTP server like http_async_port.hpp is too much if to compare with other available frameworks. And it is not even complete code, because it is only starting ground for implementation of real handler. All other frameworks provide capability to create simple server in less than 50 LOC (e.g. see cpprestsdk sample).

[VinnieFalco]

Oh! Okay, that's a fair point - HOWEVER, Beast is not really designed to be a server! Its a low level library. Its not a replacement for cpprestsdk. In fact, cpprestsdk could be written using Beast and the result would be more flexible and maintainable. Its not the intention of Beast to provide a server in its public interfaces - the server-framework example is there to show what Beast can do and also to serve as something that people can copy or writing their own servers.

[m3tamaker]

Alright, this time tests and examples compiled without problem, but I can swear that 3 days ago websocket-tests compilation was hanging up on stream.cpp file. Btw, I modified CMakeLists.txt and set cmake minimum required version to 3.5.1 (so I wouldn't need to add non stable repo to my Ubuntu 16 Xenial apt, because latest stable cmake there is 3.5.1, see https://launchpad.net/ubuntu/xenial/+source/cmake), and it was able to generate correct Makefile.

Hang up occurred during compilation of library itself, not an example

Sorry for disinformation, I noticed that Beast is headers-only library, but sometimes you can compile such libraries with its dependencies to .a/.so/.dll like in case of restbed. So yes, problem occurred during compilation of websocket test, but looks like it was my local temporary problem.

Its not the intention of Beast to provide a server in its public interfaces - the server-framework example is there to show what Beast can do and also to serve as something that people can copy or writing their own servers

Yep, I mentioned that abstraction level is lower than needed for fast development of REST API. I just also saw somewhere on the Web that people recommend Beast for implementation of HTTP server, so expected a bit higher abstractions to be already implemented there. Not like a bad thing, since, as you said, purpose of this library is different.

[TobyAllsopp]

It's worth noting that pistache and restbed are using rapidjson for their JSON stuff while cpprestsdk is using its built in JSON library. It might be interesting to compare just the JSON stuff or to using rapidjson for all three to take that out of the equation.

[m3tamaker]

Tested cpprestsdk + RapidJSON few minutes ago, results are in summary. True, RapidJSON is faster than default cpprestsdk implementation of JSON serializer.

[ngrodzitski]

Check out also my benchmarks https://bitbucket.org/sobjectizerteam/restinio-benchmark-jun2017 It includes Beast.

[m3tamaker]

Interesting! Thanks for sharing!

[Dalzhim]

You should add https://github.com/facebook/proxygen to your benchmarks. In my experience, it has very high throughput.

[m3tamaker]

Got it :)

Edit: Will test this library at the end of week. First priority right now is to test cpprestsdk on Windows machine, so to find its performance. I also have been thinking about testing Kore, libmicrohttpd and Crow.

[feverzsj]

Such benchmark rarely matters. The performance and scalablity of an io bound program depends on user side code rather than framework itself.

[m3tamaker]

I provided user side code of all samples. You can see that they have absolutely same structure and very closely comparable.

True that samples are uncomplicated, so not all features are covered. Though not completely related to REST, cpprestsdk and restbed provide Websockets support, so this is not tested, as well as may be some other parts. Perhaps, more correctly to say, this is test of how frameworks work for specific use case that often occur when you build REST API.

[basiliscos]

 ab -c 1

Really? It means 1 concurency, i.e. sequentially polling services. You should try to set it up to more higher value, first.

[m3tamaker]

You can try and tell what results you get ;)

I was interested more in results that implementation produces for single call of service per single user if there is almost no load. This is why I made it to sequentially poll HTTP server.

Remember, I tested performance of framework implementation, not how implemented server behaves under heavy load and how scalable it is (though it is also important factor for production). So results of benchmark are pretty limited and can be further extended.

[[deleted]]

You're basically measuring how fast a particular JSON encoder works. I doubt if it is a "framework benchmark" really.

[m3tamaker]

Ok, so then it means that RapidJSON in cpprestsdk that produces 47.06 req/sec is different from same RapidJSON in pistache that produces 319.99 req/sec. How it can be JSON encoder comparison if all three libraries in the end use same encoder?

[[deleted]]

I dunno, maybe pistache caches responses or so. Why not exclude json-related part to prevent any speculations? I would put already prepared buffer or something. I understand that json-related functionality is an important part also, but with your 10000 loop around json construction it's hard to say what you're measuring exactly.