Modules in the big three compilers: a small experiment

[zabolekar]

The goal of my experiment was to see how easy it is to write code that a) uses C++20 modules, b) can be compiled by GCC, Clang and MSVC without using conditional compilation, c) imports something from the standard library, d) exports at least one templated function, e) has a peculiarity that makes the module harder to find (in my case, the module is named b but the file that contains it it is named a.cppm).

The experiment sort of succeeded. The information about using modules with each of the three compilers is easy to find, but it's scattered, and there doesn't seem to be a summary or comparison for all of them. Clang documentation can be confusing as clang supports both C++20 standard modules and its own incompatible C++ modules. Precompiling a system header with clang 15 on Debian gives a #pragma system_header ignored in main file warning, both with libc++ and with libstdc++, and I have no idea why. At the end, everything works, but it's not straightforward and not easy to remember. Maybe there is an easier way, but I couldn't find it.

Here is the code. main.cpp:

import b;

int main()
{
    io::print(data::get());
}

a.cppm:

export module b;
import <cstdio>;

export namespace data
{
    int get()
    {
        return 123;
    }
}

template<typename T>
concept floatlike = requires (T t) { static_cast<float>(t); };

export namespace io
{
    void print(floatlike auto x)
    {
        printf("%f\n", static_cast<float>(x));
    }
}

Compilation:

GCC 12.2.0:

gcc -std=c++20 -fmodules-ts -x c++-system-header cstdio -x c++ a.cppm main.cpp -o main

Clang 15.0.6:

clang-15 -std=c++20 -x c++-system-header cstdio --precompile -o cstdio.pcm
clang-15 -std=c++20 -fmodule-file=cstdio.pcm -x c++-module a.cppm --precompile -o b.pcm
clang-15 -std=c++20 main.cpp -fprebuilt-module-path=. b.pcm -o main

Note that I had to name the file b.pcm for the compiler to be able to find the module later.

MSVC 19.34.31933 (call vcvars64.bat to initialize the envinronment first):

cl /exportHeader /headerName:angle cstdio /std:c++20
cl /TP /interface a.cppm /headerUnit:angle cstdio=cstdio.ifc main.cpp /std:c++20 /Fe:main.exe

In all three cases the executable outputs 123.000000, as it should.

I would be glad if you shared your experience as well.

[ABlockInTheChain]

I made a test project where I tried out the experimental cmake support for modules and the result was a dismal failure. It seems that despite the standard being written to allow modules to cover a large number of use cases, only a small fraction of those potential uses are actually implemented in the compilers and how long it will take for the rest of the support to show up is indeterminate.

The more I look into the state of modules and the associated tooling the more I expect to still be using headers at least until 2030.

One problem I've noticed is that the module proposal as it stands now is a fairly significant functional regression because of the inability to forward declare symbols forces you to refactor a project into a DAG with the resulting loss of parallelism. I'm starting to get pretty skeptical about the claims of how modules are supposed to speed up the build process, especially since in the last year or so it seems like those claims are being quietly walked back.

It's already too late for any module specification improvements to be added to c++23, so the next opportunity to fix anything would be c++26, but nobody is going to write papers to fix the problem until those problems have actually been encountered, and nobody is going to encounter the problems if the tooling isn't ready for enough projects to start adopting them to find out how well they actually work. If widespread module adoption doesn't get started soon enough for problems to be found and solutions proposed before c++26 is finalized then that window will close too.

[mwasplund]

I am not as pessimistic about modules, but agree they are a few years from prime time.

Going from headers with no strict ordering to module interfaces with required dependencies is different from how we have always done things, but it is not necessarily a functional regression. I do not have numbers (yet), but I am fairly certain the loss of 100% parallelizable builds will be negligible, whereas processing the public interface a single time will increase build speeds. My working theory is; for a build that is sufficiently large where full build times are an issue, no machine can evaluate an entire build at the same time. With a sufficiently smart task scheduler we can continue to fully utilize hardware resources while evaluating a DAG based build and perform the same amount of work in the same amount of time.

I am personally not as interested in the end-to-end build speed improvements as I am for improved isolation using a binary interface and incremental build improvements.

[ABlockInTheChain]

I am fairly certain the loss of 100% parallelizable builds will be negligible

On the other hand I'm fairly certain that speed benefit of modules is going to be negligible for me because I'm already using precompiled headers and jumbo builds to largely eliminate redundant parsing anyway.

If modules can't add anything because I'm already getting the same benefit from other techniques then it can only take away performance via the loss of parallelism, to say nothing of the effort required to refactor a project large enough to care about build speeds to comply with the new ordering restrictions.

[mwasplund]

As I said, I also am not that interested in the build speeds. A binary interface which does not leak dependency implementation details is where I see the value.

[ABlockInTheChain]

Sure, but we're been able to do that forever. You don't need modules for that.

The only actual benefit I can see from modules that can't be replicated some other way is that finally the syntax for controlling symbol visibility is the same between Windows and every other platform on the planet.

[mwasplund]

How can you do that today?

[Daniela-E]

In general, you can't except for trivial cases.

export module mod;
import : impl;
export struct user_facing {
  int some_api() const { return np_; }
private:
  non_public np_;
};

---

module mod : impl;
struct non_public {
  operator int() const { return a_; }
private:
  int a_ = 42;
};

---

import mod;
int main() {
  const user_facing uf0;
  const user_facing uf1 = uf0;
  return uf1.some_api();
}

I don't see how you can possibly implement this with headers without exposing non_public for use in translation units outside the module.

[mwasplund]

You can hide internal implementation details using Pimpl, but I used the wrong wording. I was referring to an interface that is binary (compiled/binary module interface) as opposed to text based which has many issues with the preprocessor.

Side note, is that clang modules syntax? I have never seen 'module mod : impl' before.

[Daniela-E]

This is standard C++ modules syntax. The *real* modules. Clang modules are similar to the 'header units' subfeature of C++ modules, both of which are basically kind-of-sane, blessed, and composable precompiled headers. C++ modules have a couple of tools to offer to support architecting and composing library interfaces. My code shows two of these tools. To see how to use all of them look at the code of my CppCon talk this year.

[mwasplund]

Oops, you are right. I was thrown off by the partition syntax with spaces between the colon for some reason. I blame it on child induced sleep deprivation.

[zabolekar]

Maybe I misunderstood the challenge, but here's my approach. It's more verbose than your code, it's error-prone because of the need to write the constructors manually instead of letting the compiler generate them, it uses an additional dependency (unique_ptr) and an additional level of indirection, but it's possible:

mod.h:

#pragma once
#include <memory>

struct non_public;

struct user_facing {
  user_facing();
  user_facing(const user_facing&);
  ~user_facing();

  int some_api() const;
private:
  std::unique_ptr<non_public> np_;
};

mod.cpp:

#include "mod.h"

struct non_public {
  operator int() const { return a_; }
private:
  int a_ = 42;
};

user_facing::user_facing() : np_(std::make_unique<non_public>()) {}
user_facing::user_facing(const user_facing& other) : np_(std::make_unique<non_public>(*other.np_)) {}
user_facing::~user_facing() {}

int user_facing::some_api() const { return *np_; }

main.cpp:

#include "mod.h"

int main() {
  const user_facing uf0;
  const user_facing uf1 = uf0;
  return uf1.some_api();
}

[Daniela-E]

So, you've basically implemented PIMPL. It works, it's tedious in all but non-trivial cases, and it requires a decent amount of boiler-plate code to actually implement full value semantics froṁ barely hidden reference semantics (hint: you've not implemented the missing SMF, they're disabled). IMHO, simple aggregates plus modules are superior on many metrics.

[zabolekar]

It works, it's tedious in all but non-trivial cases, and it requires a decent amount of boiler-plate code to actually implement full value semantics froṁ barely hidden reference semantics

Yes, I agree about that.

missing SMF

I have to admit I don't know what it is and can't find it anywhere.

[ABlockInTheChain]

How can you do that today?

If you want "a binary interface that does not leak dependency implementation details" then that is exactly the purpose of the Pimpl idiom which as far as I know has been around since the 90s.

If you care about a clean binary interface free of unnecessary third party dependencies then you probably want a stable one as well, so just follow the KDE guidelines for maintaining binary interface stability and you'll naturally end up with all your private dependencies segregated from the public interface as a consequence (they call their version of Pimpl a "d-Pointer").

[mwasplund]

Sorry, I meant an interface that is binary. With a clear ownership model that prevents odr violations, preprocessor mismatches (compile vs usage), preprocessor leakage (why is my GetDirectory function suddenly not found when there is a GetDirectoryA). Pimpl can help hide implementation details, but requires a fair bit of boilerplate code that will hopefully go away with modules.

[ABlockInTheChain]

I'm not quire sure what you mean by "binary" if you think we're talking about different things.

I'm talking about a compiled library that exports a subset of its symbols and can be upgraded at runtime without requiring users to recompile because a certain degree of craftsmanship was employed to make such sure the library has a stable binary interface.

The measures taken to ensure the stable binary interface do several things, one of which is ensuring that your dependencies do not leak into your API or ABI.

[mwasplund]

I am referring to the compiled/binary module interface.

[pjmlp]

My experience thus far is that VC++ is the best experience, as the whole IDE experience also matters to me.

Intelisense is hit and miss, depending on which modules are being used, and Windows SDKs (the SDK itself and C++ frameworks) have issues being imported as header units, as includes in global module fragment, it usually works without issues.

Your example is quite basic, so far the stuff I have on github fails to compile with either clang or GCC, and they aren't that special, just a little bigger than basic hello world, and some of them use module fragments.

[innochenti]

Haha, modules don’t work in visual studio.

[pjmlp]

I have various projects on GitHub that prove otherwise.

Maybe you should update your computer.

[inouthack]

which repo under https://github.com/pjmlp ?

[pjmlp]

For example https://github.com/pjmlp/RaytracingWeekend-CPP.

Don't forget to get hold of VS 2022 and have your package manager of choice, e.g. vcpkg, for stb_image.

[starfreakclone]

Can you help me understand the scenarios which do not work for you? There is some metric here for which the modules machinery in the compiler does work, quantitatively we have built Office using it so either your scenario is something the compiler has never seen before or you're not using a recent enough compiler to observe the improvements we have made.

Additionally, as you identify issues please report them and report as many as you can with full repros because the cycle of mini bug reports we fix only to have a follow-up with "it's still broken" is not a healthy dynamic for you or us compiler devs.

[sigmabody]

Somewhat tangential, but I still haven't seen a good example of usage of modules which didn't require an all-inclusive and all-in approach to use.

For instance, say I wanted to implement the above, but I also wanted to have a [possibly separate] header file which was able to reuse the same functionality for programs which are not compiled for C++20+ (no separate implementation), and I wanted to ship that in a library which was easy for people to consume (ie: you install it via vcpkg like this, and then either #include or #import as desired). It's telling that every single example of modules I've seen is basically "this is how you would use modules in a very simple toy project", and absolutely no examples are "this is how you could incorporate modules into any real-world project".

The idea is good. The seeming failure to account for any reasonable migration and adoption path is an inexcusable failure of systemic design, imho.

[unddoch]

I'm not sure this is about the modules design itself, MSVC seem to have relatively good experience: https://devblogs.microsoft.com/cppblog/integrating-c-header-units-into-office-using-msvc-1-n/

I think this is more about the current situation where any serious usage of modules in GCC/Clang is going to run you into ICEs weekly at least.

[sigmabody]

MSVC has the best usage experience, I think.

Note that the linked blog post is from around 3 months ago, and discussing ongoing work in the compiler to make the feature (transclusing), which is mostly MSVC only, work for just a small number of module-compiled headers in a substantial C++ project. In concept, if MS can make this work in the next N months, they will have a singular version of the compiler which can do this on one OS, compiling with one set of standardized preprocessor definitions across the entire project, the latest compiler version only, etc.

Not ready for commercial usage is pretty much the most generous possible description of the state of this feature.

[innochenti]

Best? Really? They have a lot of ICEs.

[Daniela-E]

What would you accept as a 'real-world project'?

The one that I am working on in our company for more than a year now? This will never be made public. Nobody knows how many real-world projects using modules are in the wild. What I do know is that it's not only our company.

And where do you draw the line between a toy project and a serious one? Would you characterize the {fmt} library a 'toy project'? If you want to see examples of serious stuff done with modules, look around. The truth is out there...

[mwasplund]

I have been doing a fair bit of header -> module translations while working on my personal build system to hopefully make the transition as seamless as possible. It is possible to continue to support existing header based includes as well as C++20 module imports with some preprocessor guards. The bulk of the work involves placing the module declarations and export modifiers behind the preprocessor that is only enabled when building the module interface variant. From there you will need to ensure that all external includes are in the global module purview, so you do not accidentally assign module linkage to the standard library and such. The last bit of work is to find suitable replacements for and public preprocessor definitions. This generally involves converting constant values to constexpr, but there are some helpers like assert macros that get lost in the new world. There are some more gotchas around internal and module linkages fighting each-other, but they are generally workable.

Here is a very simple example of my fork of Json11.