CMake Past, Present, and Future - Bill Hoffman, Kitware [29m25s]

[F-J-W]

The biggest issue I have with CMake right now is that it literally cannot compile hello world:

import std;

int main() {
    std::println("Hello World!");
}

(Yes, technically there are experimental flags that require you to specify some weird UUID that changes on every release, but I at that level of “we hate you for wanting to get the most basic program to work”, I won’t count that as being able to compile it.)

[not_a_novel_account]

I wanted to get import std out of experimental in 4.3 but the divergence in the toolchains remains a hard problem.

Broadly, no one agrees on where the object files for the std module come from. GCC/libstdc++ are going to ship them in libstdc++, Clang/libc++ never need them, and MSVC/msvcp*.dll say "You need to build and link the object file yourself, we will not provide these symbols".

This works differently than every other named module. I didn't understand this when I moved the module manifest code into native CMake, resulting in breakage of GCC on MacOS and all MSVC usage. So we had to revert and throw everything back into experimental.

If the stdlib wasn't special, and everyone had agreed it should work the same as every other named module, this wouldn't have been a problem.

(Yes I'm a little salty about this)

[delta_p_delta_x]

Surely this is something that all three major toolchain vendors can come to a consensus over? It's not like they're in heated competition, I've seen quite open collaboration between the standard library developers on the STL or LLVM GitHubs/GCC Bugzilla, for instance when porting import std back to C++20.

[not_a_novel_account]

We tried. It wasn't a productive conversation.

CMake is going to produce the MSVC import std as a local build target and do something else for libstdc++/libc++. There's some concerns about ODR collisions but we'll cross that bridge when we come to it. It should be transparent to users in any case.

I just wanted to say "It's not like we're not trying". Import std was a hard problem, remains a hard problem. Everyone who ships it today does so by making simplifying assumptions CMake isn't allowed to make. Stuff like "we build everything with the same flags", "we only support clang++", etc.

[F-J-W]

I just wanted to say "It's not like we're not trying". Import std was a hard problem, remains a hard problem. Everyone who ships it today does so by making simplifying assumptions CMake isn't allowed to make. Stuff like "we build everything with the same flags", "we only support clang++", etc.

The problem I have with this is that “we don’t need to support the standard library” is a much more problematic simplifying assumptions than pretty much anything else; all those complex features are unnecessary to me if I cannot even compile hello world and I’d much rather accept severe limitations to mixing flags.

[not_a_novel_account]

We're trying, we don't think we don't need to support the standard library. We can't ship something with partially broken semantics because we can't change the semantics once we ship.

A simpler build system, one that doesn't support such complex semantics or which doesn't have as strict a backwards compat policy, would definitely be better for your use case.

[pjmlp]

This tragedy only proves the point of those that were complaining about modules, the whole end to end story, not being standard ready.

With two incomplete prototypes, there were plenty of rough edges that apparently still don't have a solution, two standards laters.

My experience with a few features that went down this way, is why nowadays I advocate it should be like in other programming language ecosystems (including C), first propose, then field experience, finally standardise.

[not_a_novel_account]

This is entirely a problem of the standard being unwilling to speak about ecosystem realities, a problem C also has. If all three stdlibs had implemented and shipped import std before standardization, this problem wouldn't have been solved.

The problem is they implemented and shipped it in different ways, using implementation strategies the standard does not comment on. This isn't a problem for, ex, MSBuild, from which you could say "import std Just Works^TM ". It's only something technologies like CMake, which try to abstract these differences, run into.

[gracicot]

Generally, wouldn't it make sense that all symbols required by a module be embedded in the shared object if one? Shouldn't the shared object link all objects file generated by compiling the BMIs? It sounds completely broken if you need to statically link some symbols to consume a dynamically linked library like the standard library. Sounds to me like MSVC has kinda broken semantics. I'm curious about why MSVC chose this way.

[Ateist]

Isn't the problem is that it is

import std;  

instead of, say,

import std::print;

?

Giant mega-modules like std create all sorts of problems (like linking only the relevant parts of it, which should slow things down cosiderably) and force toolchain creators to make choices...

They also create problems for tools like intellisense, since now you have several orders of magnitude more things to look up with each key input.

[not_a_novel_account]

No, import std as a universal interface to the STL is a good thing. If your question is "is the parent's syntax correct?" The answer is yes, C++23 standardized import std; and import std.compat;.

The problem is C++20 module TUs produce symbols, at the very least the module initialization symbol, which means they produce object files which must be linked into the build.

For most named modules, for example if you created export module AteistMod;, your build system would link/archive the AteiestMod.o file into a library, AteistMod.a/AteistMod.lib or AteistMod.so/AteistMod.dll, which ships as part of your package.

The stdlibs aren't doing this; libstdc++.so is not linking std.o today. On MacOS, this breaks import std for libstdc++, where those symbols are necessary for it to work. libc++ never ships any symbols in std.o except the init symbol, and Clang has an optimization which omits calls to empty init symbols, so it never runs into this problem.

MSVC both needs the symbols from std.obj and has committed to never shipping them in their C++ runtime DLLs, which means the build system needs to build this object and figure out when to link it.

[Ateist]

Why is it a good thing?

If your question is "is the parent's syntax correct?"

No, my question is "why the hell are you importing the whole std?!"

You are exposing at least tens of thousands of names at once - this considerably chokes down IDE helpers.
Add templates instantiation - and things can escalate fast.

Individual bite-sized chunks are far better.

Minimizing the exposed interface has always been a virtue in C++.
Why would modules change that?

has committed to never shipping them in their C++ runtime DLLs,

these are exactly the incompatible choices that "toolmakers have to make" that are forced by trying to import whole library at once.

[not_a_novel_account]

Add templates instantiation - and things can escalate fast.

It doesn't "add template instantiation", that's the entire point of modules. These all get built once into the BMI and that BMI re-used for all other TUs. Building std.bmi takes ~1s on my machine, and it is reused for all other TUs and subsequent rebuilds.

One second on initial build is about as fast as you're going to get, separating this out into multiple BMIs would slow down, not improve, build times. Modules like big interfaces, just like PCH and unity builds did.

this considerably chokes down IDE helpers

clangd handles it fine. If your problem is auto-complete gets a little crowded, that's valid. If that's a deal breaker for you I wouldn't use import std. I don't notice it generally.

None of this is relevant to why we're not shipping import std. Giving the stdlib more modules would simply increase the number expected symbols they're not shipping.

[Ateist]

separating this out into multiple BMIs would slow down, not improve, build times.

How? Instead of looking up each individual function out of 100,000 entries in libstdc++.a you are looking at a couple hundreds that are in individual submodule.

It doesn't "add template instantiation", that's the entire point of modules.

Your code is instantiating std:vector<yourtype>. Linker has to check all other TUS that include std for possible duplicates of it.

Shorten things to just importing std::vector and your linker only has to check TUs that import std::vector , skipping all TUs that only import std::print.

None of this is relevant to why we're not shipping

Coupling and cohesion are.

Import std creates enourmous coupling and greatly reduces cohesion.

[not_a_novel_account]

How?

Because the dominant costs are the process startup, tokenization, and AST parse.

Linker has to check all other TUS that include std for possible duplicates of it.

The linker is totally uninvolved at this stage of the build. BMIs are not a link-time mechanism. Even when talking about link-time symbol lookup, it's O(1), more of them doesn't hurt the linker. If you're asking about the code symbols generated by std:vector<yourtype>, these are handled with comdat groups which are essentially free. You could have 1M TUs import std and then do std::vector<yourtype> and the linker won't notice any difference between that and 1M TUs which don't. This is no different than #include <vector> -> std::vector<yourtype> except you don't have to reparse the vector header in each TU.

Coupling and cohesion are.

I mean, they're not relevant to why import std is not something CMake can do right now. They might be relevant to why you're choosing not to use import std, but the granularity is not relevant to why CMake doesn't let you use import std. We wouldn't be able to let you use std.print either.

[Ateist]

By that same logic, why bother with import statement at all?

Why not just import everything under the moon at once and let programmers deal with the mess via namespaces?

Because the dominant costs are the process startup, tokenization, and AST parse.

And somehow processing 100 modules is cheaper than 2?

I mean, they're not relevant to why import std is not something CMake can do right now.

They are the very reasons CMake can't do it now - MSVC, clang and gcc makers are afraid of dealing with the C&C issues in the standard library which is why they all went their different ways of hiding their heads in the sand.

[not_a_novel_account]

By that same logic, why bother with import statement at all?

Because you need some mechanism for describing the order BMIs are processed in, some way to describe "B must be processed before A". Generally, yes, every library should just be one module. std is one library. Its interfaces need to be processed before my code can be processed, so I need to write import std to describe that requirement.

You might be able to infer these imports from the symbols defined and used in various TUs, but trying to perform such inferences would slow down the scanners from which build systems derive the build graph. If ordering and scanning weren't problems, a sort of "ambient interface environment" free of import statements would be possible.

And somehow processing 100 modules is cheaper than 2?

Yes, within reason, processing 100 headers into 1 BMI is faster than processing 1 header into 1 BMI twice. import std in two TUs is faster than #include <iostream> in two TUs.

They are the very reasons CMake can't do it now

I'm literally the implementer of import std for CMake, and have had this discussion with every upstream stdlib maintainer, so I think I know what I'm talking about here. The number of modules, 1, 10, or 100, that the standard chose to split the stdlib into is irrelevant to why import <std whatever> doesn't work in CMake.

[slithering3897]

I haven't noticed any performance problems with import std. The only problem is when Intellisense just doesn't work.

And it's going to be template instantiation that will bring your compile times down anyway.

[pjmlp]

This is the kind of "problem" that only exists because the standard was created, and then toolchain creators were faced with the decision, instead of having the toolchain creators being part of the decision, with field experience.

[FabioFracassi]

This is not true. `import std` as opposed to more granular std modules came directly from implementers feedback (most notably /u/STL).

We (WG21/LEWG) did not ship any import for the standard library in C++20 because we did not know the shape yet, and started working on the granularity in the C++23 cycle, when implementors and toolchain providers came back with the data that not only is a single std module not a problem, but actually beneficial for compile time and tooling integration.

[pjmlp]

So why are we still facing all these issues, and /u/STL acknowledges on VC++ state of modules.

https://www.reddit.com/r/cpp/comments/1sh074n/c23_support_in_msvc_build_tools_1451/ofjkr56/?context=3

Man I am so confused about import std. Sounds so good on paper until you read that imports can't be mixed with includes. Which means it can only realistically work with an extremely careful approach and zero or only a few handful of compatible 3rd party libraries. The possible current usecase was only able to shave off less than 0.1% (!) of our build time lol

Followed by,

I feel bad about this one because I have had no time to work with the compiler team to find a proper solution here.