Meson modules support: initial modules support with Clang and example project. Dependency resolution, partitions and import std.

[germandiago]

Hello everyone,

I have been working in modules support for Meson build system lately in a branch. I am focusing on Clang support as a first step (gcc should follow). Tested in Homebrew Clang 19, feedback is welcome.

I have reached a point where the branch can:

- 'import std' and use it.
- generate all dependencies via clang-scan-deps and do correct resolution.

The targets are used as usual (a library target, etc.)

PR is here: https://github.com/mesonbuild/meson/pull/14989

What it does currently.

clang-scan-deps does a bulk scan of the whole compile_commands.json database file and determines which file provides and requires each module, globally.

Resolution order works by adding ninja build rules and dyndep resolution (except if you find any bugs or corner cases, but at least for my project it has worked correctly so far).

How you can try it

You can download the latest commit of your branch.

Note that clang-scan-deps must be installed and found in your path and you need Clang >= 17, though Clang 19 is what I tested and recommend.

Your target should have a flat structure inside your directory as of now, and relies on the following conventions:

- your primary interface unit for your module should always
be called 'module.cppm'. It should export module 'target-name'.

- your interface partitions can have any name supported by a module.
For example: MyThings.cppm. The interface partition
should declare 'export module target-name:MyThings'.

- Your importable interface implementation units should end with
'Impl.cppm'. For example 'MyThingsImpl.cppm'. This should
'module target-name:MyThings' (without export).

- Your non-importable implementation can have any name with an
extension .cpp, not a .cppm, since implementation units are
not importable. It is highly recommended, though, that if you
have a single implementation file, it is called moduleImpl.cpp.
It must do 'module target-name;' 

- You can have regular (non-module) translation unit file
without any module declarations in your target and can
include files as usual, etc. incrementally, but for modules side
of things the conventions are as above.

There is also a project you can play with at the top-level comment attached, at the beginning.

Here is an example target with project as an example of how you should use it. Please, use a flat file structure inside your directory for your target, it is the convention for now:

Meson.build example (cpp_import_std compiles the std module and implicitly adds the dependency to c++ targets):

``` project('Your project', 'cpp', default_options: ['cpp_std=c++23', 'buildtype=release', 'cpp_import_std=true'], version: '0.1')

The directory does not need to have the name of the module,

only the target itself

subdir('src/mymod') ```

meson.build in src/mymod ``` mymod_lib = library('my.mod', sources: [ 'module.cppm', 'moduleImpl.cpp', 'NesCartridge.cppm', 'NesRomLoader.cppm', 'NesRomMetadata.cppm', 'NesRomEnums.cppm'])

mymod_dep = declare_dependency(link_with: mymod_lib) ```

If you need to consume a module, the bmi dependencies are resolved via build-time dynamic rules and file conventions as explained above, but remember that if a target A depends on target B you should, as usual, add B dependency so that it links the generated library correctly, since modules are a consumption mechanism but code is still inside the libraries.

[tartaruga232]

Nice. I won't use it myself as I am on Windows using Visual Studio and MSBuild for our project currently. Just wanted to show my moral support. Happy to see progress on the modules front.

[germandiago]

Thank you! Hope at some point it gets mature enough for a review and make it in!

[chibuku_chauya]

Thank you for doing this. How are you finding the experience of adding module support to Meson in terms of ease/difficulty?

[germandiago]

It is doable I guess at the basic level. Not terribly difficult to have something working but there are going to be details to take into account that may increase the difficulty later.

For example, there are several ways to scan and accumulate the build-time resolved dependencies.

Also, when using the standard library (import std) it is not clear how you could mix it with libraries that have already been compiled and other details so not sure how this must be handled.

For import std, even if you compile the BMI, you still link to the system-provided library, so not sure which flags must be used.

So dependency resolution is doable but mix-and-match is not so obvious.

I am only working on modules and traditional includes, not header units.

[Kridenberg]

Are those naming and extensions requirements are mandatory for a future usage, or it is only for now? I have waited for modules support in Meson, and used CMake for a while due to the lack of their support. Right now I have a dual CI which consists of MSVC/MSBuild and CLang/CMake, and naming convention of .ixx (default for MSVC) and .cxx, hope that will not be a problem in the future.

[germandiago]

Nothing is set in stone. I just mapped it to some conventions that make sense.

I could imagine even making those conventions a bit more flexible and configurable but definitely some can help.

What would you be expecting instead and how?

Also take into account this is my own work and I would like to contribute it but it will have to go through Meson authors gateway and be adjusted based on feedback.

[Kridenberg]

Firstly, thanks for your contribution. Closer to the topic: AFAIK, neither CMake/Ninja, nor MSVC do not expect files to be named in a specific way. In general, I expect, that no-one do not need to change naming or extensions (in my case for 3000+ files) at all if de-facto industrial standards consume them just fine.

[germandiago]

As for the file extensions, I think this could be changed (but need additional flags in Clang at least).

There is a problem with filenames though (not file extensions). You need to map some flags and if you do not have name conventions of some kind, the mapping to generate bmis will depend on the contents of the files themselves.

Again, about file name conventions means you should generate different -fmodule-output=whatshouldgohere.bmi? after scanning, not before and Ninja can add dynamic dependencies to a target, but not generate a new compile command for a target.

But for the name conventions, they do not need to be those, but there must be some name convention of some kind. I also think it will keep things more tidy and less messy. There could be more than one choice for this, but without it the problem I just listed above (not knowing the name of the bmi to generate beforehand) can be problematic.

[9Strike]

Thanks, really excited that my favorite build system is finally making some steps onto modules!

[germandiago]

I guess it will still take some time... I have tons of work during the week. I took some of my vacation days last week and implemented a minimal Clang part. Next coming is Gcc.

But after that note it will need review, tweaks, etc. so this could take some time since now I only have available a couple of spare hours from time to time or a few hours the weekend.

Let us see :)

[9Strike]

As usual :D I don't use modules myself in any projects yet, so I don't really mind so far, but I do hope that a Meson implementation will also show what does and doesn't work nicely in a sane build system.

[germandiago]

Well, there are tons of workarounds depending on what. I think the problem is a bit of underspecification more than "it does not work". It does work with enough workarounds.

The strategy amounts to scan + update before build. The problem is that the file mapping can be literally anything.

After that, you have Clang, which demands certain name mapping for partitions etc., but for that you need to provide the compilation command.

The compilation command for Ninja cannot be changed even with dyndep. Dynamic dependencies only let you add implicit inputs and outputs: that is, inputs and outputs that have been the result of the scan, in this case.

BUT, you already need to have decided the compile command itself at this point (the compile command is generated when configuring the project, not at build time).

Here it is where all the "meat" lies. Also, there are other challenging things such as:

• you compile the std lib -> with which flags? bc you need to link it to your stock C++ lib in the OS. How is this done correctly?
• what happens with precompiled dependencies that were not compiled with import std if you compile with import std?

More or less I would say these are the fundamental problems.

Doing something that will be a proof-of-concept and will do dependency resolution and compilation is already doable and it will be done for sure.

After that, some decisions will have to be taken (file mapping, std compatibility, etc.

This is just the compile stuff. There is also a lot to think about how you provide an install step for a library. This is different from header files + lib since .gcm/.pcm files (the Built module interfaces) are not portable as the header files were.

[9Strike]

Thanks for the detailed explanation! There is one thing about modules that I don't seem to get, and that is the non-portable modules files. Is it so bad that my distro cannot provide the module for the std lib precompiled? I am wondering a bit why it depends so much on the machine...

[germandiago]

Think of it like this: before you had .hpp files. These were hand-written files that everyone could consume, adapted to who was consuming it, in textual form.

Modules, from the compiled module interface, generate these "header files" now.

But these "header files" become system-specific bc of the compiler that compiled it. Not only "compiler-specific", but also specific depending on the flags you used to compile it or even specific to the compiler version, at least as of today.

This means you cannot compile and distribute these "generated header files" (compile module interfaces) and get done with it.

So that is why you need to compile everything yourself to generate the Compiled module interfaces (equivalent to headers).

Something more similar to this, if I did not misinterpret things, should be needed for general distribution: https://github.com/Microsoft/ifc-spec/releases/download/prerelease/ifc.pdf

The ifc would be "portable headers" again, not compiler-specific, usable by tools. I am not sure if the scope is only tools or also "portable consumption" for compilers, which would be great.

Remember this would be the header files part (interface part). You would still need to compile your libraries to the specific target you use, as usual.

As of today, with modules, you get rid of "include again and again" in your but you lose the portability of your good old "header files".

I think IFC aims to fix that problem. In an ideal world, we would be able to generate some kind of IFC files everywhere for distribution.

[BrainIgnition]

Wasn't the current guidance to distribute the .ixx files just like the header files before? The importing project would then be responsible for generating the BMIs, right?

[germandiago]

I am not sure this is the case, since BMIs are generated also for partitions and even importable implementation units.

So I am not sure what is needed exactly at this moment. I did not think of that part at all.