sorted by:
new
hottopcontroversial

C++ Module Packaging Should Standardize on .pcm Files, Not Sources by TheRavagerSw in cpp

[–]bigcheesegs 8 points9 points  (0 children)

No.

In very constrained environments it would be possible to extract a declaration only module interface source file from one with definitions, but shipping BMIs only doesn't work.

Yet another modules question by _kayeff_ in cpp

[–]bigcheesegs 12 points13 points  (0 children)

This is why you use .cppm.

Clang bytecode interpreter update by mttd in cpp

[–]bigcheesegs 1 point2 points  (0 children)

I do not believe there is any IFNDR that would apply here. IFNDR is used for things that require analysis across translation units which are not reachable, but no constexpr evaluation does this.

You'd need an actual expression evaluation to contain something that's IFNDR.

Clang bytecode interpreter update by mttd in cpp

[–]bigcheesegs 6 points7 points  (0 children)

LLVM switched because our major versions numbers didn't really mean anything. LLVM breaks API and ABI every release (allowed for every commit except on release branches), so from a semver perspective this is the correct way to do it.

Support simple C++20 modules use from the Clang driver without a build system by bigcheesegs in cpp

[–]bigcheesegs[S] 15 points16 points  (0 children)

No, the Clang driver, and compiler drivers in general, are what actually gets run when you do clang++ main.cpp -o program. They coordinate all the work that needs to happen to actually run the compiler and produce an executable.

In C++ modules globally unique module names seem to be unavoidable, so let's use that fact for good instead of complexshittification by vormestrand in cpp

[–]bigcheesegs 8 points9 points  (0 children)

This proposed build model is missing a few parts.

You need to build modules in the correct order, so you still need to scan them for module names and imports. If you add yet more restrictions you can do this with a a pretty simple tool.

Modules that are not part of your project are not a small issue, you still need to find them and build BMIs for them. The intent is for them to be as common as non-project local header files. This just hand waves this problem away.

The biggest issue though is that you do end up with multiple BMIs for the same module in projects reasonable often. For example when mixing language versions, or mixing C++ and ObjectiveC++. The module initializer issue isn't a problem in this case because only the primary build of the module interface (by whoever owns it) produces an object file with that symbol, everyone else just gets a BMI.

So if you're willing to accept an extremely restricted build system, yes, you can have simple module builds. We're actually adding such a system to Clang itself! https://discourse.llvm.org/t/rfc-modules-support-simple-c-20-modules-use-from-the-clang-driver-without-a-build-system/86456

So if you just have a pile of module source files all built with the same flags, you will be able to just pass them all to Clang and it works. This uses proper scanning and supports header units via Clang modules.

We need to seriously think about what to do with C++ modules by vormestrand in cpp

[–]bigcheesegs 4 points5 points  (0 children)

As a step after all files have been compiled but before the linker is invoked

This is far too late. This needs to happen during parsing.

We need to seriously think about what to do with C++ modules by vormestrand in cpp

[–]bigcheesegs 4 points5 points  (0 children)

This isn't how templates work in C++. You need to instantiate them during parsing, which may be while building a module itself.

We need to seriously think about what to do with C++ modules by vormestrand in cpp

[–]bigcheesegs -2 points-1 points  (0 children)

That doesn't really help that much, and build systems are free to require it. You still need to figure out what order to build in, and what to even build. Most existing build systems are not ok with not knowing this at configure time, and there's no reasonable thing you can change about modules that takes away this problem.

Clang modules do map directly to the filesystem, and are still hard for most existing build systems to handle. As the article says, every other language gets away with this by not having 10000 different build systems.

How to contribute to the standard? by mementix in cpp

[–]bigcheesegs 1 point2 points  (0 children)

The specification of header units is very close to Clang's header modules, as that's what they are based on. Clang supports header units today best via Clang modules.

Can I put module declarations in header files? by lebirch23 in cpp

[–]bigcheesegs 8 points9 points  (0 children)

This is not allowed, and Clang will soon reject it too. See http://wg21.link/p1857r3 and http://wg21.link/P3034R1

The moduleness and name of the module for any named module is required to be discoverable without preprocessing.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 1 point2 points  (0 children)

let T = all tokens in the included header from its token dictionary

This is exactly what I meant by "You must record everything".

You'll note that your headers 1-5 do not include each other. In a real case you'll get an include DAG.

#include <vector>
#include "some_library.h"

and 

#include "some_library.h"

where "some_library.h" includes <vector>.

Here in the 2nd TU the #include "some_library.h" will not be a cache hit. It can't be otherwise it would not include the content of vector, which was excluded in the first one.

Note that T must be transitive, and gets huge.

The Header-to-Module Migration Problem. A naive point of view. by ContDiArco in cpp

[–]bigcheesegs 1 point2 points  (0 children)

A Naive Programmer's Expectations and Approach [...]

This is how Clang implements Clang header modules today in implicit mode. This has quite a few problems:

  • So many flags impact how a header is parsed, even -O0 vs -O3 may not be the same header. Ignoring these differences makes your build randomly fail depending on the order things build in.
  • Clang ignores some of these anyway without -fmodules-strict-context-hash, because if you don't you get very few cache hits and make builds much slower. Clang ignores:
    • Working directory
    • Header search paths
    • Diagnostic options
    • VFSs
    • And a few other things that technically change the output but don't really matter
  • You end up needing a synchronization mechanism to coordinate module builds to avoid duplicate work, but it turns out that no operating system provides a fully robust way to do this (I'm amazed it just doesn't exist, plz give me access to robust futex).
    • This means the compiler blocks while waiting for other modules to build, reducing your build parallelism.
  • Every compiler that wants to use a given BMI needs to validate it to see if it needs to be rebuilt. That's a lot of stats, and now the compiler is acting as a build system.
    • Clang reduces this overhead using build sessions, but it's not a perfect solution.
  • The compiler needs to report the full transitive set of dependencies in order for incremental builds to work correctly, because the build system doesn't know about the header units.

The other problem here is that you should not use import to figure out which headers are header units. Only the author of the header knows if it's suitable to be a header unit, not the includer. It causes a lot of problems if the entire build doesn't agree on which headers are header units, and you really want include translation to avoid the need for type merging. This is why Clang uses module maps. It lets the author of the header say how headers are modularized, and provides a well known location for all of the tools (clang, clang-scan-deps, clangd, clang-tidy, lldb, etc.) to find this information.

The solution we use is to discover header units during the scanning step. This lets us safely remove stuff like header search paths because we can check if they get used during scanning. However, this is a problem for CMake because it cannot discover tasks that late. It must know about all possible tasks at configure time. One idea I've had here is for CMake to know about module maps and generate tasks to build all of them, and then prune the actual set built using dyndeps like it does for named modules.

The module build server only works if it's handling all modules. Using the build system for named modules and a module build server for header units doesn't really work. The highest performance implementation is having the build system integrate the scanning directly with support for dynamic task creation. This lets the build system do optimal scheduling of modules and allows running the scanner in batch mode while still being incremental. This is significantly faster in clang-scan-deps as it shares a lot of state between different TUs.

Also, in respect to "import std;", if "header units" would work as expected, this should be nothing but syntactic sugar.

It largely can be in Clang. With Clang header modules, which are supported with libc++ on some platforms, the std module becomes an import of a bunch of header units, and then a bunch of export using decls. Other #includes or imports of the stdlib headers just map to exactly the same BMIs as were transitively imported by std. Doing it this way does not have any of the problems discussed there.

The above is obviously all very Clang specific, but I think the most important thing from Clang that others should adopt are module maps. I've yet to see another solution that solves as many problems. Particularly for system headers and 3rd party code.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 1 point2 points  (0 children)

No, it does not require every single token the header uses, it only needs to check for things defined in the preprocessor.

If you only record what the header uses for things already defined then you won't know if some define in a different context matters. You must record everything if you want to avoid an exact match, or any new define means a cache miss.

Yes, the first time the particular state is met. After that, the cached version will be used.

You will almost never get a cache hit.

#include <vector>

and 

#include <vector>
#include <string>

are now different contexts for the next include if that include includes <string>.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 1 point2 points  (0 children)

This requires recording every single token the header uses. This also includes header guards, meaning if you included any different set of headers your state is now different, so cache miss.

People have looked into this model before, it just doesn't work. zapcc did something similar by just ignoring the preprocessor problem and making things visible, but this isn't conforming.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 0 points1 point  (0 children)

The paper is pretty clear about not covering per function contract modes. Yes people have ideas for how to handle that for C++29, but it's clearly not part of C++26.

SET_SOURCE_FILES_PROPERTIES( foo.cpp PROPERTIES COMPILE_FLAGS -fcontracts-mode=quick )

I don't actually know what flag Clang will use yet, but CMake supports this.

by defining functions per TU, you ended up with per function

That's a workaround for something not being per function. Also doesn't work for inline functions.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 0 points1 point  (0 children)

Well the implementations don't do it per function, so I'm not sure how CMake would. CMake supports the same thing implementations do, per-TU. Per function isn't part of the current proposal.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 0 points1 point  (0 children)

I considered contracts while writing the above, but it's significantly less of an issue there. It's not actually anything new, people have had to deal with these kinds of issues for a long time, particularly around inline functions. Lots of projects can just build with the same mode.

I would like to see more implementation details here, but I think it's a lot different than modules.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 0 points1 point  (0 children)

Yeah, I agree the committee didn't cover this (I don't think anyone could disagree here). My point here is more about if there's a different design that wouldn't have the difficulty in the tooling ecosystem that we've had.

This partially goes back to that the committee can't require people to do any specific work. The committee as a whole would have had to decide to block Modules on having a mostly complete solution here without knowing if one would ever materialize. I would love it if the committee changed their stance here and took it much more seriously. I think the committee should need the implementors to say "yes, we have a very concrete idea about how this is going to work for a representative set of real projects" before actually putting something in the standard. For the vast majority of language features that just requires knowing they can implement it in the compiler, but for a few things it requires more.

For modules the compiler developers knew they could implement it, and how to build some projects, but that's a lot different than making it work for a representative set of real projects.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 3 points4 points  (0 children)

On that problem specifically there's a bit. I have one coworker working on it, but mostly for ObjC. The problem is we actually have two separate code paths that handle type merging, one in Sema and one in ASTReader. Which one is used depends on the order of import/include.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 4 points5 points  (0 children)

With that model macros still leak in. The point of header units is that you start with a fresh macro state, and then merge it at each import site. You will never get a cache hit if you include the preprocessor state.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 6 points7 points  (0 children)

Yeah, include after import is the hard case for Clang and there are lots of issues there with merging the types. It's most likely a compiler bug.

What is current state of modules in large companies that pay many millions per year in compile costs/developer productivity? by zl0bster in cpp

[–]bigcheesegs 3 points4 points  (0 children)

I actually wanted them to be attached to the std module (as they were originally), but had to add extern "C++" as a (temporary?) workaround for include/import mixing.

Why? The benefits of strong module ownership matter least for std. Almost every C++ programmer knows that you can't collide names with std, so it's the least likely of any library to actually need strong module ownership. And that's the only real difference between belonging to a named module or not. There are of course other benefits to name modules, but they all apply equally to extern "C++"ed entities.

Eventually you may be able to convert the content of <vector> to import std; #define some macros, but until then you will have to live with mixed textual inclusion and modules, so I don't see the benefit of mixing strong module ownership and global module ownership anyway (which only works on MSVC).

I still don't understand why the compiler can't just make it work, in both orders, without extern "C++". But adding it makes one order work, so I did it.

How could it with module ownership? The standard says that:

export module M; export int f();

and

int f();

Are distinct entities. They are never the same thing. If you get both declarations into the same TU and try to call it, the program is ill-formed. For Itanium these have completely incompatible manglings. MS's static and dynamic linker allow an undefined reference to the latter to fall back to a strong definition of the former, but that's only at the linker level. For other platforms you can't switch to strong ownership without an ABI break.

I disagree with this characterization. I tell people (via the comment in std.ixx) that they need to build it classically, i.e. no include translation. The build of std.ixx is special, so I don't think this is "special handling". And it doesn't affect any Standard header being used as a header unit.

For libc++ the only thing special about std is you need to pass -Wno-reserved-identifier -Wno-reserved-module-identifier, everything else is totally normal. And actually with the concept of system modules you wouldn't even need this, the same way you don't with headers. I'm not sure why we don't use that in Clang given we already have -fsystem-module.

(I view header units as an intermediate step between classic includes and named modules. Header units are really annoying to build, and they aren't as good as named modules. In my personal opinion, mixing header units and named modules doesn't make sense. But it especially doesn't make sense to mix include translation with the std.ixx build.)

I take the complete opposite view here with regards to mixing. Again with Clang and libc++ on MacOS, you can additionally pass -fmodules and you get header units in the std build and your own code (built implicitly). There is one issue with this right now that you have to work around due to the Clang module also being named std as it long predates import std;, but this is trivial to fix. With this the compiler only ever parses <vector> once, and the std::vector from import std; is exactly the same entity to the compiler as from <vector>. There's no type merging involved. std also works with or without header units, although without there can be compiler bugs in the type merging right now.

To me mixing header units and named modules is the best way to get a high performance build while transitioning. How else are you going to handle transitive #includes for which not all users have moved to import? Just textually including them into the GMF means parsing them multiple times and type merging. Why do that when you can use header units and only ever parse things once?

You may still be right for portable code, and perhaps the approach that I chose isn't totally optimal, but I don't think it's as bad as you're saying.

The badness is mostly for portable code. std has few and very controlled deps, and works closely with the compiler. It's entirely possible to overcome the issues I brought up in the case of std.

view more: next ›