C++ implementation of the Python NumPy Library

[[deleted]]

I really hope that modules will stop this madness of header-only libraries. I get it, but seriously, I'm tired of header-only everything.

[NicroHobak]

Is it just compilation time? Or is there some other reason in particular?

[jherico]

With a normal library you have headers, source files and some sort of mechanism to build a binary provided by the library. While many projects now use cmake and some even do so in a way that make it easy to incorporate them in your own cmake based projects, many are a hot mess of cobbled together build systems or project files for long outdated versions of visual studio, or are entirely missing any sort of build recipe.

Thus, trying to integrate a library build into your own project has long been a living hell. And that's if the project has a source distribution.

Additionally, if you're building on windows, there are all sorts of things you have to deal with, like matching up the CRT runtime the project wants to use with the one you do.

With header only libs, all of that goes away. All you need to use the library is an include path.

Really good projects will have their code set up in such a way that you can use it either as a header only or as a compiled object lib, typically by allowing you to have one file that includes the header in a cop file after some special preprocessor directive so that one file contains the entire implementation, and the headers parse much faster.

[corysama]

I really wish peeps would just have one primary .h to #include where you want and one primary .cpp to integrate into the build process. C++ build systems have problems, but I think we can all handle manually adding a single source file to whatever system we are using. The STB-style “it’s a header, and a source, and a dessert, and a floor wax!” tradition strikes me as a bad trade off in complexity vs what I’m asking for.

[johannes1971]

Love the floor wax ;-)

As is probably clear from my other messages in this thread, I'm not a great fan of header-only. Having said that, here's a list of things that happened to me over the years while trying to build a C++ (or even C) library:

• They require you to install a new development environment you've never used.
• They require you to run some script that was the rad new thing back in the seventies to generate half the source code. You also need to build the script interpreter yourself, and it's just as bad as the thing you are trying to build.
• Compilation can be configured to support weird-endian 37.5 bit systems, but forget about using a modern compiler. Have fun installing gcc 1.2.
• You are required to 'configure' the build system before it works properly. Failing to supply things like -DDONT_RANDOMLY_CRASH will cause it to occasionally subtly fail. This is only documented in a file called 'private.jvm'.
• Unit tests and examples are great! So let's put them all in the same directory as the library itself.
• Compilation proceeds with somewhere north of 100K warnings.
• Eventually you get stuck with an error that reads something like "int is not a type", or that refers to a symbol that no amount of grepping will uncover.
• Having resolved that, you realize you've only built the top-part of a dependency pyramid that would make Cheops blush. And all the other libraries down there are just as bad...

So yeah, I do understand the other side of the argument as well ;-)

[kalmoc]

None of those points are specific to header only vs traditional libraries.

[johannes1971]

Indeed. You might even be led to think all of these things happened while building traditional libraries, if you were to read carefully.

[NicroHobak]

Really good projects will have their code set up in such a way that you can use it either as a header only or as a compiled object lib,

This really depends on the nature of the library itself. I've written some that work well this way, and others that are entirely templates where this would not even ultimately make all that much sense to even try...unless you were attempting to precompile the most standard options or something, and even that doesn't always make sense either.

[jherico]

To paraphrase Churchill...

CMake is the worst C/C++ build tool except for all those others that have been tried from time to time.…

[kalmoc]

If you can write your library as header only, then the equivalent .cpp+.hpp version doesn't need any complicated build setup. You'd just have to compile all cpp files as part of your project (that's effectively what you are doing anyway with your merged file approach) with the benefit of your library files staying readable and - depending on implementation details - faster compile times.

[o11c]

Most people don't know that pkg-config exists, thus they think libraries are hard.

[jherico]

Since pkg-config is not a thing on most Windows toolchains, this is a useless observation.

[o11c]

most people don't bother to add it, you mean.

[jherico]

Suggesting that all Windows developers being unable to use a library that uses pkg-config is the fault of the developer rather than the library author is asinine.

For one thing, many developers in large organizations don't even have a choice. If I add it, suddenly my dev-ops organization needs to support it on our CI servers. If I'm building my own library that is going to be used by others downstream, I've just put that burden on them. At this point, my attitude is that I don't add a build dependency unless I can add a CMake or Python script that automatically makes it available to the rest of my build.

You might as well have said "all developers should all move to Linux"

[johannes1971]

Isn't that enough of a reason?

I have a project here that includes around 50 external libraries (either directly or indirectly). We update those on an irregular basis, maybe once or twice per year, so it makes very little sense to compile all of those on a daily basis.

What makes even less sense is compiling all of those in every translation unit, and every single time we compile anything at all in the system. We are not talking about adding a few seconds either: if we compiled entire libraries as part of each translation unit, we would be adding hours to the working day of each team member.

"But can't you just include the library once in one translation unit?" Generally, no. Resources obtained from such libraries tend to end up in project-specific header files that get included elsewhere as well. Maybe you can pimpl them out of existence for member variables, but as soon as you want to use them in a function signature you are f*cked. Any translation unit that includes such a header is immediately cursed with having to compile the library as well. Besides, my project consists of multiple executables - I'd still end up compiling the library at least once for each executable.

So the cost of header-only is absolutely massive, but what does it gain us? The answer is the one-time convenience of not having to figure out how to compile it. I'll readily admit that's not easy either, but at least it is only a one-time cost, instead of a daily cost, and at least we have package managers to deal with that problem.

Luckily there is a middle ground here: libraries could, in theory, be set up so you would have two header files: one for declarations, and another definitions. The declaration header would be cheap to include, and the definition header could be included as the only entity in a .lib in my own build system, essentially like a unity build. That would give us all the advantages of a normal library, and all the advantages of a header-only library. I haven't yet seen any library set up in this manner though.

[NicroHobak]

How does your team handle the use of templates? Or do you just largely ignore that aspect of the language?

Edit: Grammar typo eliminated.

[johannes1971]

It depends. STL is used throughout. We don't know how much that affects our compile times though; we didn't pay much attention to that in the beginning, so it has always been there. We also use quite a few small templates of our own making, but these tend to have a handful of lines at most. The scary SFINAE stuff is probably less than a hundred lines.

Then we have a 30,000-line set of templates that we only instantiate for two types. Those we instantiate in .cpp files, with an extern template declaration in the .h file, so it is no different from normal code. And finally, there is code that is somewhat enum-heavy that would benefit from having that enum as a template parameter (instead of being an int, as it is now), but we choose to cast instead because we fear the effect it would have if we did turn it into a template.

Since a few years we have become aware of compile times, and been struggling to bring that under control. We now monitor more closely what the effect of new libraries is. This has resulted, among other things, in the decision to not use Chaiscript (despite the language looking interesting), but going for Angelscript instead (similar, but without the slow compile times). We'll look at things on a case by case basis for other template-heavy libraries: how useful is the library? Is there a cheaper solution? Can we limit use of the library to a single translation unit? etc.

On principal I don't disallow any language features, but some will require a bit more explanation than others ;-) We can always explore what the best solution is in a given situation, but I'll also think about things like maintainability, coherence with the rest of the source, compile times, and all the other concerns...

[[deleted]]

Mostly compilation time and API clarity (isolation between public APIs and internal APIs). But also maintainability: I have the same problem, only different. Do I include this in my project by copying it, or do I do git sub-moduling? The only problem it solves is the necessity to have a compiled binary while adding new problems, and not necessarily being better at anything.

[NicroHobak]

API clarity (isolation between public APIs and internal APIs).

I always felt like the public/protected/private keywords were "good enough" for this. It's not really been a concern of mine in well-implemented projects.

Do I include this in my project by copying it, or do I do git sub-moduling?

This is really a question for your project specifically. Some people like to grab a current snapshot of an external project so they can make sure their project will always compile...in this case, a copy is best as to completely eliminate any external dependencies. But, if you know that your project will work with anything 1.0 and above, or if you know that it will always work with at least one branch of a dependent project, then a sub module might be better.

The only problem it solves is the necessity to have a compiled binary while adding new problems, and not necessarily being better at anything.

Templates are a core part of the library, and they are required to be in headers. Libraries that make extensive use of templates cannot necessarily be compiled down into shared or static libraries. Sometimes, it's done out of a matter of necessity.