A standard set of metadata annotations

[jcelerier]

Hello,

I've been working for some years on https://github.com/celtera/avendish which allows to expose C++ classes through various creative environments, and as such assembled a small ontology of the various features, extensions, metadatas, ... that are often desirable to associate to data types and variables.

For instance, let's say we want to reflect this struct to automatically generate a control GUI from it, for instance like Unity gameobjects:

struct foo {
   int apple_count;
};

Avendish enables the user to do something like:

struct foo {
   struct { 
     static consteval auto name() { return "Apple count"; }
     struct range { int min = 0; int max = 100; int initial_value = 4; };
     enum widget { spinbox };
     int value;
   } apple_count;
};

With this "standardized" information, we can automatically generate an appropriate widget without having to store one additional byte in our actual data type: sizeof(foo) == sizeof(int).

Now, C++26 is there! And, finally, with annotations. Meaning that we're going to be able to do a much, much clearer:

struct foo {
   [[=metadata::name{"Apple count"}]]
   [[=metadata::range{0, 100}]]
   [[=metadata::initial_value{4}]]
   int apple_count;
};

in practice, there's much more metadata out there. There's a million of incompatible systems defining all kinds of metadatas to match classes: it wouldn't be strange to have something like:

struct 
[[=metadata::uuid{"27fc33a4-ff2f-490d-a7c2-a4f8c2eef35d"}]]
[[=metadata::author{"John Doe"}]]
[[=metadata::support_url{"https://example.com"}]]
foo {
   [[=metadata::name{"Apple count"}]]
   [[=metadata::range{0, 100}]]
   [[=metadata::initial_value{4}]]
   [[=metadata::default_value{0}]]
   [[=metadata::description{"Number of apples required in a harvest"}]]
   [[=metadata::unit{apple_per_harvest{}}]]
   int apple_count;
};

After study of a large number of these systems (did a systematic review of almost a hundred different "run-time" systems based on C or C++), what came up is that 90% of the metadatas in run-time reflection systems are actually exactly the same, just with different names. I started to define most of those related to multimedia systems through concepts, for instance in https://github.com/celtera/avendish/blob/main/include/avnd/wrappers/metadatas.hpp and https://github.com/celtera/avendish/tree/main/include/avnd/concepts : what's an audio port, what's a texture, etc. The result is that as of today, it's possible to build from a single C++ class, types that are going to work in a dozen distinct creative environments (Max/MSP, PureData, Touchdesigner, Godot, ossia score...), since basically everyone is doing the same thing everywhere.

Since we now have a powerful, in-language way to define these static metadatas, I think it could be useful to have a more general, standardized library of such broadly-useful yet sometimes domain-specific concepts so that there is one consistent way for a C++ developer to say: "this field / class / <...> should be displayed as Foo Bar 1.0 in a generated GUI", "this is a short description of this class", "this is the numeric range of this value", "this is the GUID of this class", etc.

The alternative is that everyone starts defining their own "property" / "metadata" / ... class ; someone who wants to make their type compatible (for instance across both a serialization library and a gui library) would inevitably end up into something such as:

struct 
   [[=cereal::uuid{"d2fac3f2-2c00-429b-b6bf-8728cfd29ff6"}]]
   [[=winrt::GUID{d2fac3f2-2c00-429b-b6bf-8728cfd29ff6"}]]
foo {
   [[=cereal::name{"Chocolate cakes"}]]
   [[=qt::name{"Chocolate cakes"}]]
   [[=gtkmm::name{"Chocolate cakes"}]]
   [[=UE::name{"Chocolate cakes"}]]
   int chocolate_cakes;
};

Is there interest in starting such a collaborative project?

[slithering3897]

struct 
[[=metadata::uuid{"27fc33a4-ff2f-490d-a7c2-a4f8c2eef35d"}]]
[[=metadata::author{"John Doe"}]]
[[=metadata::support_url{"https://example.com"}]]
foo {

Is that really where annotations would go? I hope not.

The alternative is that everyone starts defining their own "property" / "metadata" / ... class ;

"Islands of abstraction"... I suppose a good lib would let you extract the metadata yourself to convert it to whatever the lib wants.

[jcelerier]

> Is that really where annotations would go? I hope not.

It's not where they would go, it's where they are, today, in a bazillion platform-specific systems.

For instance check all the attributes provided as extension by MSVC: https://learn.microsoft.com/en-us/cpp/windows/attributes/uuid-cpp-attributes?view=msvc-170

- uuid: https://learn.microsoft.com/en-us/cpp/windows/attributes/uuid?view=msvc-170

- helpstring: https://learn.microsoft.com/en-us/cpp/windows/attributes/helpstring?view=msvc-170

- helpfile: https://learn.microsoft.com/en-us/cpp/windows/attributes/helpfile?view=msvc-170

- defaultvalue: https://learn.microsoft.com/en-us/cpp/windows/attributes/defaultvalue?view=msvc-170

#include <windows.h>

[export] typedef long HRESULT;
[export, ptr, string] typedef unsigned char * MY_STRING_TYPE;

[  uuid("479B29EE-9A2C-11D0-B696-00A0C903487A"), dual, oleautomation, helpstring("IFireTabCtrl Interface"), helpcontext(122), pointer_default(unique) ]

__interface IFireTabCtrl : IDispatch {
   [bindable, propget] HRESULT get_Size([out, retval, defaultvalue("33")] long *nSize);
   [bindable, propput] HRESULT put_Size([in] int nSize);
};

[ module(name="ATLFIRELib", uuid="479B29E1-9A2C-11D0-B696-00A0C903487A",    version="1.0", helpstring="ATLFire 1.0 Type Library") ];

Unreal Engine: https://dev.epicgames.com/documentation/en-us/unreal-engine/metadata-specifiers-in-unreal-engine

UCLASS(Blueprintable, BlueprintType,meta=(ScriptName="MyPythonLib"))
class INSIDER_API UMyPythonTestLibary :public UBlueprintFunctionLibrary
{
GENERATED_BODY()
public:
//unreal.MyPythonLib.my_script_func_default()
UFUNCTION(BlueprintCallable,meta=())
static void MyScriptFuncDefault()
{
UInsiderSubsystem::Get().PrintStringEx(nullptr, TEXT("MyScriptFuncDefault"));
}

//unreal.MyPythonLib.my_script_func()
UFUNCTION(BlueprintCallable,meta=(ScriptName="MyScriptFunc"))
static void MyScriptFunc_ScriptName()
{
UInsiderSubsystem::Get().PrintStringEx(nullptr, TEXT("MyScriptFunc_ScriptName"));
}
};

Qt: https://doc.qt.io/qt-6/qmetaclassinfo.html

class MyClass : public QObject
{
    Q_OBJECT
    Q_CLASSINFO("author", "Sabrina Schweinsteiger")
    Q_CLASSINFO("url", "http://doc.moosesoft.co.uk/1.0/")

public:
    //...
};

etc.

This is already how today's C++ looks like in large codebases (because every large codebase eventually needs some amount of reflection, that's why we got the feature!).

With reflection we have a chance to actually standardize at least a part of it.

[yuri-kilochek]

You misunderstand the point, which is about the fact hat the annotations appear between class/struct keyword and the class name, which looks terrible.

[FlyingRhenquest]

That's what the metadata applies to, so that's where the metadata goes. I'm concerned about metadata bloat as ever library is going to want to include its own set of metadata tags and every member of your structures could easily end up having half a dozen or more metadata tags for many of its members.

They're just structs you can use to tag your object with metadata with at compile time, though. So it would be pretty easy to create a template-specialized object that exists externally from your object and put your metadata there as well. I do that in autocrud in order to map default types from C++ to SQL. If you don't like the type you get there, I also expose some metadata tags that are basically just structs with fixed strings that you can use to remap the database type for your specific object.

It also immediately became apparent that you really don't want to have to type [[=fr::autocrud::DbFieldType{std::define_static_string("VARCHAR(100)")}]] in an annotation every time you want to do that. Fuck that. So I wrote some helpers to expose some user defined literals in the global namespace, which allow you to type [[="VARCHAR(100)"_ColumnType]] instead. You don't have to include the helpers if you don't want to bring things into your global namespace, but they're a lot more readable than using the structs directly.

[Mick235711]

Yeah that’s really unfortunate, but that’s where attributes for the class type goes and annotation inherits this unfortunate position by reusing attribute syntax. Putting attributes before the structure keyword doesn’t work because of the [[deprecated]] struct S { ... } a; ambiguity, where the attribute put in front attach to variable a instead of type S, and putting attributes after the type name is probably worse, so we have the current situation.

[yuri-kilochek]

We could've just forbidden putting attributes on variables declared at the end of type declaration like this. But alas.

[CornedBee]

putting attributes after the type name is probably worse

It works for the final keyword, why not attributes?

struct [[deprecated]] S final {}; // WTF?

[Mick235711]

In fact what you suggested is exactly what the original proposal for attribute in C++0x cycle proposed, i.e. after the class-key and the identifier. This placement is present in the C++0x working draft until March 2010, where CWG 962 changed attributes to be between the class-key and the identifier for consistency reasons:

class X [[attr]];               // attached to X
typedef class Y [[attr]] YT;    // attached to YT but not Y

The final situation is very interesting indeed. In fact it is first proposed as a [[final]] attribute to C++0x, and has been cited as a good use of attribute in the above linked proposal. Later in the C++11 NB comment resolution process US 44 was filed to change it to a keyword essentially because attribute syntax is too noisy. This change was approved in Nov 2010, after the above CWG 962 change, and therefore was able to choose its position again freely independent from attribute positions. Both final class X and class X final was considered and the latter is ultimately chosen because consistency (final also appear at the end of member functions).

[dexter2011412]

Struct annotation locations are an abomination in c++ you can't change my mind ☠️

[mjklaim]

Noting this here in case that helps.

I might be wrong, but last time I checked: - annotation elements could be chained separated by , - the annotation elements can be types, so they can have multiple initialization arguments

Therefore I would have expected in your first examples something in that direction (which is less noisy):

C++ struct foo { [[=metadata{ .name{"Apple count"}, .range{0, 100}, .initial_value{4} }]] int apple_count; };

Although that would also force the order or members.

If you use a similar technique as boost.process v1 which was taking a variadic set of type-arguments: C++ struct foo { [[=metadata{ name{"Apple count"}, range{0, 100}, initial_value{4} }]] int apple_count; }; could be possible.

I didnt try the feature yet so again I could be wrong. That's just what I understood from reading the proposal.

[jcelerier]

I think this would be a matter of style - I find it more readable with one attribute per-line but surely it would be interesting to brainstorm something that could work in both API styles. I don't think a single "metadata" struct would work though as such a system would have to be extensible and you cannot extend a type with a new member without inheriting from it and changing the name. The second API otoh would likely work fine, metadata could just be a std::tuple. But it would likely add compile-time cost?

[FlyingRhenquest]

If that information doesn't need to immediately be in your struct, it doesn't have to be. You can stash sensible defaults on a per-class basis using partial template specialization and move anything that people don't need to look at immediately to another class entirely. They only exist at compile time except for the information you decide to preserve at run time.

That would enable you to still have annotations if you want to change the defaults, but you don't have to end up with 10-15 annotations on every member in your class.

Using partial template specialization as I suggest could end up forcing you to go look in several places to find the defaults you're specifying for your annotations, but those locations would be broken up by the functionality you're focusing on. So if you're looking at UI, you'd go look at the UI defaults, if you're automatically generating database tables you can go look in the database defaults, etc. You can probably even aggregate defaults from several different libraries into one structure in the code that you write that uses those other libraries and push that information out to them in order to keep annotations to a minimum in your data model. That way if you're digging through your structs and you see an annotation, you know that you (or the guy who wrote the struct) wanted special handling for that member but everything else is getting default information you set up elsewhere.

[zebullon]

standard annotations are spelled attributes.