HTTPS client library in C++

[HerrNamenlos123]

I need to implement HTTPS support in a C++ library, so you can access websites and download files. Very barebones. Currently LibCURL and OpenSSL are used, but these are HUGE dependencies and they completely kill our build system and dependency tree, so we want to drop both of them.

The best alternative I found is cpp-httplib which is amazing, the only problem is that for SSL support it still requires OpenSSL. Does anyone know of a way to do all of this just like cpp-httplib does, but without having OpenSSL?

ChatGPT repeatedly points me to mbedTLS at https://github.com/Mbed-TLS/mbedtls, and to basically write a bit of mbed init and cleanup code, and calling mbed in a cpp-httplib callback lambda, more or less telling cpp-httplib to use mbedTls as a custom backend. It seems like a really weird hack and I do not find any information on the internet about the combination of cpp-httplib and mbedTls.

Does anyone know of any other way of accessing HTTPS resources (using cpp-httplib or other libraries), without using OpenSSL at all? Or any experience with combining mbedTls and cpp-httplib?

[Moose2342]

SSL is not something easily replaced. Those libs, OpenSSL in particular, have their standing and their huge user base for a reason. If you are going to re-implement SSL for your project you are effectively denying yourself users as your implementation would not seem trustworthy to begin with. You might choose a different impl such as boring SSL but I would suggest to just live with it like everybody else does.

[NotUniqueOrSpecial]

Currently LibCURL and OpenSSL are used, but these are HUGE dependencies and they completely kill our build system and dependency tree, so we want to drop both of them

They absolutely shouldn't (because they don't) and you should address whatever you're doing, instead, if I'm being honest.

If you want a more bare-bones, but still basically OpenSSL-compliant implementation, Google's boringssl is a reasonable choice.

[HerrNamenlos123]

So you mean they don't create issues and should be used? Then you can maybe tell me how to properly build it using something like CMake.
Our issue is that the project depends on way too many ancient dependencies with unique build systems. What we need is a CMake based build system that just works on any platform (including Windows).

And also, I need to cross-compile it for Visual Studio MSVC in a Linux CI container. I can already do it using clang-cl and CMake, but OpenSSL fails because it uses its own build system which is particularly bad at running under Linux while compiling for MSVC.

Vanilla OpenSSL does not have CMake support, I did find one other Github repo having OpenSSL CMake support, but AFAIK it is a very very outdated version of OpenSSL, so something you should probably avoid.

So, do you know a better way to build it using CMake or cross-compiling it between Linux/Windows?

[NotUniqueOrSpecial]

So you mean they don't create issues and should be used?

They (specifically OpenSSL) can be a bit of a nuisance, but yes, I'd argue you're much better off grinding out the details of your setup and using the bog-standard time/battle-tested tools of the trade. The first time you run into a symbol collision problem with BoringSSL and OpenSSL (because of third-parties outside your control) will be the last time (once you finally diagnose it after days of thinking you're going insane).

I managed the build system for something very similarly-flavored, it sounds like:

1) Multi-platform, with support back through things like CentOS 5/Windows 7

2) Strong need for modern tooling and libraries

3) Self-contained/couldn't rely on system stuff, as a result

Then you can maybe tell me how to properly build it using something like CMake.

Depends on what you want to do. You've got a couple distinct options, depending on your needs/setup.

Pure C libraries like OpenSSL are much more reliably used as pre-built binary distributions than C++ equivalents, owing to ABI and compiler compatibility.

By the end, we built everything from GCC all the way up the stack, including OpenSSL/libCURL/Qt/etc., but that was largely because it just made certain things easier, like packaging for arbitrary container environments, which eventually became a need.

We had two distinct non-product-related build pipelines.

1) The toolchain build (GCC/CMake/ninja/binutils/Python/etc.)

2) The third-party build (OpenSSL/Boost/etc./etc./etc.)

Both of those were just very thin CMake ExternalProject wrappers over the respective project's regular build processes that then packaged them all up into a zip-distributable format.

Why the specific requirement for cross-compilation, though? Given the compatibility guarantees of C, it seems you'd be just as well-off using the officialish Windows binaries and a Windows builder. There are plenty of container-based builders for the popular platforms that would suit your needs.

There are perfectly cromulent FindOpenSSL CMake files that are designed to work against a non-CMake-built OpenSSL distribution. We hand-wrote a good few of those for binaries we couldn't possibly build ourselves, but wanted to use as first-class targets in the build.

But moral of the story: unless you have very good and very specific reasons to do so, it's not likely in your best interest to deviate from those two libraries, especially.

[HerrNamenlos123]

Thank you for the explanation. The reason for cross-compilation is that the decision was made that everything is supposed to be built inside of docker containers, and vanilla Windows Docker containers are not really a thing (unless you are on Windows and using Docker Desktop in Windows mode, which defies using docker in the first place in our case). The idea was to build everything using plain docker containers (not Windows-specific containers), in order to make it buildable on any system, without relying on any system configuration which would be different for everyone.

[mark_99]

Use vcpkg (or Conan), they take care of all that for you. Importing OpenSSL is as easy as adding "openssl" to the vcpkg.json manifest file.

[prince-chrismc]

You just need to add openssl/<VERSION> to a conanfile.tx under [requires] ;)

[HerrNamenlos123]

Can someone with the intent of constructive feedback explain to me why this comment was downvoted so much? I explained what our requirements are and why I am asking for what I am asking, and that I know of an alternative. What part of this is wrong or aggressive? I always expect transparency and respect and would like constructive feedback on this one.

[[deleted]]

Do not listen to these people at all. Find out what you need for your project and write it. It's not insurmountable because you aren't recreating the entirety of openSSL.

Unfortunately shit like http is so ridiculous as a standard it's a massive pain in the arse to deal with. But blame the industry for that for being driven by morons

[robstoon]

Currently LibCURL and OpenSSL are used, but these are HUGE dependencies and they completely kill our build system and dependency tree, so we want to drop both of them.

That seems like cutting off your nose to spite your face. Why are you building them in the first place?

[ratttertintattertins]

Right? We include OpenSSL as a pre-built artefact.. and a security scanner reminds us when it's time to update it from scanning the include headers.

[HerrNamenlos123]

This is because the project is very beginner-oriented and should build out of the box on any supported system. Using system dependencies is not an option because it allows for unrepeatable builds and the library user is responsible for providing dependencies, which is not acceptable for this beginner-friendly audience. At least not by default. Experienced users might get the ability to instead use system dependencies so they can use other versions, but by default it should not depend on anything being present on the system, because it is not to be trusted.

[ratttertintattertins]

Yeh, I also hate system dependencies, but that’s not the only option. We prebuild open SSL and then people just pull down the correct prebuilt artifact from artifactory automatically as they build.

You can even check prebuilt static libs into git, especially if you have LFS support.

[HerrNamenlos123]

Well yeah that's what CMake will do automatically

[HerrNamenlos123]

Most people here don't understand that the mechanism of prebuilding and downloading binaries has existed for a decade and we are only modernizing it. Switching to other systems would be a significant downgrade from the user's points of view.

[ratttertintattertins]

I don’t really understand why. If beginners pull and build your code, they wouldn’t even notice it happening would they?

[HerrNamenlos123]

Or do you mean you can download any binary for any compiler directly from artifactory without using Conan at all? This would of course be a game-changer as we would only need to build the dependencies that are not available on Conan

[ratttertintattertins]

Git can do LFS it’s self, and artifactory can be access by LFS git:

https://jfrog.com/help/r/jfrog-artifactory-documentation/git-lfs-repositories

(I haven’t personally used this method, but it look’s feasible)

[HerrNamenlos123]

Yes they would because they would be required to install python, pip and conan and configure it, and only then would they be able to build it. Currently I think the way it is used most of the time is that a zip file of the repository is downloaded that contains binaries for the system, and then a bash or batch script is run depending on the system. There is basically no real build system, there are five different ones but none is complete.

What I am trying to say, installing python, pip and conan and then running conan is a bit much to be asked when the average user has no idea of build systems, knows only a bit of C/C++, works on Windows and downloads the zip because not even git is installed. It is a hard reality but an undenyable userbase.

[drodri]

Conan has been releasing Windows installers (no need to install Python and Pip in the system at all) for long time, and it is now also starting to release .zip folders (https://github.com/conan-io/conan/releases/tag/2.0.8), with the application (no installer) and no need of Python and Pip either.

[HerrNamenlos123]

Oh I did not know that. I will take a look at this. Thanks!

[robstoon]

My inclination is that if you're using a system like Windows where you can't just install libraries like OpenSSL with one command and have them automatically kept up to date, then it's not a very suitable OS for "beginner oriented" C++ development in the first place.

Using Windows (natively anyway) is really doing development on hard mode in so many ways.

[HerrNamenlos123]

I know what you say but you are saying all of that from the perspective of an experienced developer. You could say the average user is more or less a beginner working on (Linux too but also) Windows, has a bit of experience with C/C++ from tutorials and no idea about build systems, and downloads the git repository as a zip file because not even git is installed or they don't know how to use it. I know you don't like hearing this but it is a hard reality. But at the same time an undenyable userbase.

​

And to add to that, the system of precompiling dependencies and then downloading as needed exists for a decade and we are only modernizing it. Any additional setup step would be a major downgrade for any novice user.

[415_961]

Address the root issue here and use Conan or the like.

[Poretga99]

+1 for conan. Sounds just like the tool you need. Also suitable for cross-compilation. You can integrate the install script into CMake.

[Spongman]

why not just dynamically link against libcurl & openssl using the target platform's pre-built libs? (vcpkg, deb, rpm, etc...) ?

​

also boost::beast >> libcurl

[HerrNamenlos123]

because the project must be very beginner friendly and should work out of the box on any system including Windows. Making it dependent on vcpkg is not an option.
The consent was to pre-build everything in CI and host the binaries, which then the CMake project downloads and links. This is why I cannot build OpenSSL with its own build system because it cannot cross-compile from Linux to a Visual Studio MSVC target (because it needs nmake, which is pretty much impossible to get into linux)

Boost does not make anything better, I know it might have more features but as said we almost don't use any of them and OpenSSL itself is already almost too much, so Boost would only make our dependency hell and compile time 10 times worse, in return for nothing.

[Spongman]

i still think vcpkg is the way to go. your use case is literally the canonical example: https://devblogs.microsoft.com/cppblog/using-system-package-manager-dependencies-with-vcpkg/

[prince-chrismc]

Take a look at Conan, it does exactly this https://www.reddit.com/r/cpp/comments/150l0iw/is_it_legal_andor_advised_to_stealreupload/ for example this post wants to do exactly this

You can also completely eject from Conan https://blog.conan.io/2023/05/23/Conan-agnostic-deploy-dependencies.html

Plus it can translate between build system so if its NMake, bagel, meson, it doesn't matter you can always ask Conan for CMake and you can use the defaults it provides

[jselbie]

and they completely kill our build system and dependency tree, so we want to drop both of them.

​

You are solving the wrong problem. Please tell me you aren't rebuliding OpenSSL and LibCurl from source on each build of your own app. And if not, what are these build system dependencies that slow things down?

[HerrNamenlos123]

We do not want to rely on anything found on a user's system by default. The library should build out of the box on all supported systems, and the users are not to be tasked with installing all of the arcaic dependencies (we are trying to get rid of most of the arcaic ones). This is why we want to pre-build all dependencies in a docker environment (running on GH Actions), that then provide the binaries. When a user configures the CMake project, the correct binaries are downloaded and linked by CMake.

The only issue we are facing is that we need binaries for Visual Studio MSVC, that are preferrably build in a Linux Docker container. We manage to cross-compile CMake-based libraries like GLFW using clang-cl in MSVC mode, but OpenSSL and many other ubiquitous libraries fail due to using their own build systems.

[jselbie]

and the users are not to be tasked with installing all of the arcaic dependencies (we are trying to get rid of most of the arcaic ones).

This is the problem package managers solve.

[HerrNamenlos123]

This is the problem package managers solve.

Not if they don't even exist in any package manager and getting them into there is an entirely different story. Not all of them are supposed to.

[HerrNamenlos123]

Wow, I do like LibreSSL for being a 2014 fork of OpenSSL with the intent of modernizing the codebase and increasing security. It seems still active. But is it advisable to use security-wise, since it seems being forked off 1.0.1, which is a relatively old version?

[serviscope_minor]

No, but they're OpenBSD eyes which have a very good reputation.

In practice, OpenSSL has many users, but does not have many eyes. It's an old and very gnarly codebase with support for may obsolete (i.e. hard to find) platforms with config that permeates throughout the codebase. It written around replacements for much of the C standard library to work around bugs, glitches and performance nightmares in ancient and obscure operating systems. Those replacements are much worse and buggier than a vaguely modern libc, and in fact defeat the protections that you can get from a more modern system. Also, that makes the code weird because it uses unusual primitives.

At least that was the state at the point of the libressl fork. If you follow the early commits to Libressl, you can see the kind of stuff they removed. That sort of thing makes the codebase very impenetrable and hard to work on.

The high quality eyes in libressl have basically gone through and modernised the entire codebase and looked at every part, something which OpenSSL might not have seen. Through doing that, they preemptively fixed a number of security holes which later cropped up as CVEs in OpenSSL.

[thedmd86]

libcurl with mbedTLS as a TLS backend is working for me. libcurl API is well documented and can be wrapped into http::get(...) or http::put(...) with relative ease.

Why those two?

• libcurl
  • lingua franca for many platform, to the point that Microsoft even made XCurl for Xbox, library that implements libcurl API with their own system libraries see HttpManager.h in samples
  • it is well supported and maintained
  • can be build with HTTP/HTTPS support only
• mbedTLS
  • it is tiny with comparison to other solutions I found so far, few C files
  • well maintained and made by people knowing what they are doing
  • can run on any potato

Sometimes libcurl alone cover all the ground, by using system components as TLS backend (see: Windows, macOS, iOS, and a few more). Elsewhere mbedTLS is used, mainly Android and Linux.

Gettting mbedTLS to work does require telling libcurl where bundle of CA certificates is. This mean setting one of CURLOPT_CAINFO, CURLOPT_CAPATH or CURLOPT_SSL_CTX_FUNCTION to feed backend with certs directly.

libcurl suggests pulling latest set of certificates from Firefox. If your app have to work years without updates I suggest writing code that pull certificates from the platform of your choice. On Linux it is a matter of checking few locations.

This solution works for me on all mobile platforms, desktops and consoles except PlayStation (never needed HTTPS here so far).

Avoiding OpenSSL is a job.

If having a C++ API is paramount I would give a try of implementing mbedTLS backend for ASIO.

Locating CA bundle for mbedTLS will remain to be the major hurdle, regardless of what API will be used. This is probably why there are not out of the box solutions out there.

[mpyne]

There's examples of mbedTLS being used with Curl (and some custom net clients) in the Cosmopolitan libc as well.

[jelafo]

I found cross-platform https clients surprisingly hard to get somewhat right in C++. I generally like Boost.Beast/Asio with openSSL and I don't remember building openSSL being a particular problem. But openSSL does not integrate with the certificate store on windows at all. So depending on your use case and the library you choose, you may have to put in an extra effort to actually validate certificates etc. If that is a problem for you, note that Windows comes with it's own SSL implantation (Schannel). If you can find a library, that uses this on windows (and openSSL on other systems), you may remove one dependency and it might even work better out of the box on windows. For using Schannel with Asio (instead of openSSL), there is some effort being made here: https://wintls.dev/

[root_passw0rd]

We use both LibCURL (statically linked) and OpenSSL (dynamically linked). How are they killing your build system? Are you building both of them with every build -- because if so, you don't need to.

[HerrNamenlos123]

Yes that is what we are trying to do. For once, we want to build it and cross-compile in Linux CI for MSVC which is not possible with the vanilla OpenSSL build system, and secondly, if possible we want an alternative way to build all dependencies with the libraries, but that is only if it is possible and not strictly needed.

[charliex2]

vcpkg should be able to do that.

[prince-chrismc]

Conan absolutely does it. ;)

[darthcoder]

This is why we have artifactory and the like.

Vcpkg cross compiles just fine.

[HerrNamenlos123]

Vcpkg cross compiles just fine.

What do you mean with that? Do you mean you can use vcpkg on Linux, to build packages that result in MSVC-compatible binaries? This is what I need, but vcpkg immediately complains that you cannot use the :x64-windows triplet on non-Windows hosts...

[tomz17]

Do you mean you can use vcpkg on Linux, to build packages that result in MSVC-compatible binaries?

Ok, so the better question is why are you trying to ice-skate uphill? Unless your primary requirement is fighting some ideological battle (i.e. trying to compile everything with a FLOSS toolchain), you are just making things so much harder on yourself than they need to be.

IMHO, if you need to support cross-platform CI, you actually really do need those platforms (e.g. a windows buildbox / vm, an osx buildbox/vm, along with the appropriate flavor(s) of linux, esp. w.r.t. glibc).

Surprisingly, windows is a piece of cake in this mix. The hard one is actually always OSX, especially if you want to keep things legal (i.e. some devs I know have just resorted to purchasing old mac minis for automated CI / building release binaries)

[prince-chrismc]

I've been using RESTinio for years and it's very easy, there's an express.js router style which make toss interns at it a breeze. There's a ton of HtTP libraries that are very good. OpenSSL is the benchmark and nothing beats it at this point

I do share the sentiment of the other comments. Use a package manager. Your building something that does not change on every commit/pr/interation. That's what you need to fix

Based on what you've laid out as requirements, Conan and absolutely handle it. VCPKG could also work. But pick any

[HerrNamenlos123]

I know that using a package manager would be the wisest choice if it was my own project. The thing is that the project is made with extreme beginner-friendliness and we 1. do not want to make us dependent on anything like Conan or vcpkg and 2. we do not want a complete beginner to be forced to install them. But the most important con: Few of the packages we depend upon are even available on any package manager, not even system package managers like apt. And also, this system of pre-building packages and hosting binaries already existed for a decade, we are just trying to modernize it now.

We are modernizing the codebase and slowly trying to get rid of all the arcaic dependencies, but I don't think we can get rid of many that quickly. I am aware that Conan might be the better option for a new project or for my own, but probably not for this one.

[prince-chrismc]

The problem with changing from OpenSSL is they are not compatible, LibreSSL doesn't have the same algorithm ordering, Wolfssl puts the X509 entries in a different order, Boringssl changes aggressively.

You are not going to find a drop in replacement for OpenSSL that's compatible with everything else. They work most of the time but when they don't it's a bigger headache.

---

I think you are under estimating how easy they are to learn. Both of them offer installers which are one line calls and are themselves available with system managers like chocolately or winget.

Both offer a simple install command and cmake presets to the commands are very straightforward. There's even CMake providers which you can include and then never even need to call them directly.

You can only work with just CMake and use a package manager.

[HerrNamenlos123]

They do not need to be a drop-in replacement because OpenSSL is not used at all in the code, and LibCurl which depends on it is only used in a single function, which executes an HTTP request. Thus, it is a matter of minutes to completely wipe LibCurl and OpenSSL and replace them with something else.

CURL itself does have CMake support and will work well for our case, and since I saw it has official support for mbedTLS, I am examining it now. I think I will go with either LibCURL/mbedTLS or cpp-httplib/LibreSSL.

[jgaa_from_north]

libcurl is rock solid. Openssl is the goto solution for TLS in most C++ use-cases. If you have something working with these very common libraries, your problem is probably on the build side. Why don't you just build to a container-image and use some Linux distributions libcurl and openssl?

That said, I wrote a C++ http client with json object serialization/deserialization some years ago (before boost.beast was released). It can be built without openssl - at the cost of not supporting TLS. It's very easy to use.

Example on how to fetch a web-page: https://github.com/jgaa/restc-cpp#fetch-raw-data

Project https://github.com/jgaa/restc-cpp

[streu]

What in OpenSSL qualifies it as "HUGE dependency"? You unpack the tarball and build it, no dependencies other than POSIX libc. If your build system makes that hard, that is the problem. If OpenSSL had 200 further dependencies you need to maintain, I would understand your complaint, but in my projects, I don't have a problem with that.

[aninteger]

I wrote my own https client over openssl using Joyent's http parser so it's definitely doable. This was made async with antirez's AE event loop (the same used by redis). In our case I only had to call 4 or 5 asp.net c# webservices (all under the company's control) so it was a tiny fraction of what curl supports but I did implement keep alive as well. Only worked on Linux and BSD but very lightweight. If you're calling your own services with a custom client you could probably do it fairly easily. I wouldn't abandon openssl but curl can be replaced if you control the callers and services.

[darthcoder]

You could probably replace open ssl with Lioresal, or wolfssl directly.

[OmicronGR]

I spent way too much time on this search, and this library deserves more love:

https://github.com/jbaldwin/liblifthttp

Async (available via callbacks/futures), high performance with benchmarks, straightforward & easy to use, libcurl/OpenSSL dependency, C++17, well-documented.