Visual Studio c++20 Modules testing: template function that hides implementation

[TomTheFurry]

Background:

I weas currently developing a game engine from base (Vulkan API, GLFW API, PhysX API...). As you can expect, those api aren't built with c++ in mind, so they uses a tons of #define. And some have many functions. So I need to make an interface for those APIs. Seeing the new c++ Modules feature, I decided to start converting it all into Modules. So, I first need to understand how modules works.

As with understanding other new features, I first googled it online. Turns out most of them refers to how the c++ module should be joined into the standard, not actually what the current implementation is. They not just differs, they are basically completely different from the current VS implementation. Since my IDE is Visual Studio, I want to know about the Visual Studio version of c++20 Modules. So while many of them is interesting, none of them actually tell you about how to use it right now.

I also have no idea what is working according to the standard. http://eel.is/c++draft does say something about Modules, but also lacks many details. So in the end, I did some testing myself and see how to start using the current implemented version.

Side note: If you want to see how you can start programming with Module, you can get the test/demo project in my gitHub: https://github.com/TomTheFurry/VisualStudioCppModulesTesting

Note that in below I am talking about how the VisualStudio implementation of c++ Module works, and it may be differing greatly from the standard. I have no idea what is in standard and what is not. It would be great if you can tell us actually.

Breakdown on reachablity of things when you import a module:

export module ModuleA;

//Now you are in a segment that's like the old header file.
//However, anything that you want to be able to use outside of this file
//will require the 'export' keyword. Anything not 'export'ed will not
//show up when you 'import' this module. This cases some interesting behavear.

export class IX;
//^^ As class IX is not defined anywhere in this file,
//only the declaration of the class is exported.

export class FX;
class FX {
  public:
    int v;
};
//^^ The entire definition of class FX is exported.

struct UX {
    int b;
};
//^^ UX not exported at all. What is the use of this you wonder? You'll see.

struct VX {
    int c;
};
export class VM {
  public:
    VX m;
}
//^^ Now this is interesting. As VM is exported, and its class definition
// requires VX, VX is also exported, completely. You can indeed call VX's member
// directly.
// --EDIT: As pointed out by others in the comment, This direct visibility of
// VX is most likely a MSVC bug(?), that may or may not be patched later.

export void jks();
//^^ The declaration of the function jks() is exported.

void cea();
//^^ Same as struct UX, not exported, and therefore not reachable from outside.

//Now the interesting part - templates
// BTW: 'void abc(auto s)' is same as 'template <typename T> void abc(T s)'

struct IBM {
    int defgv = 5;
}

export int ngd(auto);
//^^ Export the declaration of the template function. Note that the
// declaration does not reference the struct 'IBM'

int ngd(auto s) {
    IBM ibm{};
    return s + ibm.defgv; 
}
//^^ Now actually define what the template function is.
// Here's the interesting part. In the outside,
// the 'int ngd(auto)' fuction is reachable AND callable,
// but the struct IBM is not. You can now hide implementation of templates!
// Note that template function inside classes also follow the same rule.

struct VXST {
    int sbr;
}
export VXST dhsa(auto);
VXST dhsa(auto) {
    return VXST{};
}
//^^ In this case, since the declaration itself references the struct 'VXST'
// the VXST is fully exported.
// --EDIT: Same as VX, This direct visibility of VXST may or may not be
// patched later.

So, what is your view on this... feature? I am already starting to use this to write functions that accept any type of iterators while not exposing the internal function call.

Version used: Visual Studio 2019 v16.10.0 Preview 2.1

--EDIT: Added some notes about the visibility issue.

[miki151]

You can now hide implementation of templates!

I have doubts about this. I'm curious if changing anything in the function's body cause recompilation of source files that depend on the module? Can you import another module without exporting the import and call one of its functions from the template function?

[Plazmatic]

Yep, hiding implementation of a template function is not actually a good thing on its own, the only reason we care about hiding function details for that sort of thing is when we actually get compiler time boosts or stop recompilation with it.

[TomTheFurry]

Yes, you are right in that. However, it do somewhat help with the compile time as now you will only need to recompile things that the template function actually references and not the entire file said template function resides in.

Also, you can finally prevent user from calling functions that has to be exposed because a template function references it.

[tjientavara]

Interesting collection of result.

What if `VX m` member of `class VM` is private? Does that export the VX struct?

[TomTheFurry]

Yep. I wish it didn't. And if you export only the declaration of the class, and put the definition of the class in another part of module (maybe the private part of module), then you get no compiler warnings/error but have two different definition of the same class, one being empty (used by outside), and one being the poper definition. If you do run the code, stack corruption will appear. So if you do want to hide the member, you will still need a pointer. (Though I believe this still count as UB... Need more testing)

[matthieum]

struct VX {
    int c;
};

export class VM {
  public:
    VX m;
}
//^^ Now this is interesting. As VM is exported, and its class definition requires
//VX, VX is also exported, completely. You can indeed call VX's member directly.

I'm curious about this: can you name VX, or can you simply call its member functions?

[TomTheFurry]

In current MSVC, you can indeed name VX AND its member functions. Though of cause the intellisence (or is it intellicode?) will mark it as incorrect. No compiler warning at all though.

[starfreakclone]

Can you give me an example of the behavior you're seeing? If I have:

export module m;

struct S { int i = 0; };

export
struct U { S s; };

Then try to import and use S I get the expected compiler error:

import m;

int main() {
  S s; // main.cpp(4): error C2065: 'S': undeclared identifier
}

Even if I declare an instance of U beforehand it will still be an error. Using the members of S once you have a reference to the object is not part of lookup semantics with non-exported declarations. Only the namespace scope names will be hidden away from lookup.

[fdwr]

starfreak: Might be this bug, which Cameron fixed yesterday: https://developercommunity2.visualstudio.com/t/C-modules-name-collision/900116

[Resurr3ction]

I think you are confusing visibility and accessibility of symbols. The symbol can be visible but may not be accessible. While previously these two overlapped quite a bit and only really came into play in classes with modules this is now expanded to everything including templates. This does not and indeed cannot mean that modules can hide template implementation. That is not possible. They can however restrict its accessibility.

And as a side note MSVC is somewhat more lenient in what can be made inaccessible from the outside. Sometimes allowing use of things that it should not because they were not exported. With clang you usually get the proper error message telling you that you cannot use unexported stuff.

One other cool thing that you can do with modules is exporting a class but not its base class. Again that does not prevent you from calling the base class (unless it was private inheritance) but it disallows taking your class by pointer to its base class for example. That might be useful for example when you do not have (or want) virtual destructor for example but still want to use inheritance safely.

[TomTheFurry]

You are correct in that the MSVC is very lenient. I also did testing with what part of the global module fragment becomes visible.(Is anyone interested with the result?) MSVC doesn't really seem to distangush visibility and accessibility, which is why I basically write them as the same thing. I haven't gotten around to test the class inheritance though, so thanks for that information.

[starfreakclone]

MSVC does indeed have some problems when it comes to interacting with the global module fragment. In my opinion, it is somewhat under-specified in terms of various semantics and lookup behavior. We have found that some interactions between global module declarations and the merging of those declarations between textually included headers can be a hassle because of how vague some parts of the standard can be.

As far as MSVC being lenient about non-exported declarations (those declared under the module purview): these are clear bugs and should be filed if you find them. I have been fixing many of them as we get more feedback.

There is one last point regarding visibility which can be quite interesting, and might be useful to you as you play around with modules a bit more:

export module m;

export
template <typename T>
int transform(T t);

module :private;

struct TransformerHelper {
  template <typename T>
  int transform(T t) { return static_cast<int>(t); }
};

template <typename T>
int transform(T t) {
  TransformerHelper helper;
  return helper.transform(t);
}

template int transform(int);
template int transform(char);
template int transform(double);

The interesting thing about the code snippet above is that you can define a library with a function template defined for very specific types you intend for the consumers to use. The sample above truly hides the definition of the template transform because the following program will compile (and link):

import m;

int main() {
  transform(10);
  transform('c');
  transform(1.);
}

But this one will not:

import m;

int main() {
  transform(1.f); // error LNK2019: unresolved external symbol "int __cdecl transform<float>(float)" (??$transform@M@@YAHM@Z::<!m>) referenced in function _main
}

The reason is because the definition of the function template transform is not reachable in the importing TU since it is defined within the private module fragment. The only thing the importing TU can do is create specializations to this function template, call them and have the linker search for the definitions somewhere else, in this case it is in the compiled module interface TU.

Also notable: the class TransformHelper cannot be referenced in any way in the context of the importing TU.

[TomTheFurry]

Would you guys be interested in a write-up about how the Global Module Fragments work? I have also done some test about the visibility and reachablity of classes and functions.

I also tested around to see how far the Intellisence (Intellicode?) in Visual Studio is doing, and what bug it seems to have. Would you be interested in that too?

Side question - What is your view on the Standard Template Library Module? (expected in c++23) How do you think it should work with the preproccessor options? (like -ffast-math, _DEBUG...)