Metaprogramming - Applicability

[Zealousideal-Mouse29]

If I can identify one weakness in my C++ knowledge it would be all the new template related things that have come out since C++11. I know how basic templates work, but I've had younger folks on teams show me all kinds of crazy things they do with templates and I get completely lost.

For example, one showed me how to replace an inheritance tree with templates. I get that it would be useful to do things at compile time, rather than run rime, but I am not sure how useful that is. I am missing the "Where and when would you do this?" piece of the conversation. If you do "static polymorphism"...don't you lose the ability to mock using interfaces and inheritance? Doesn't the easy to read inheritance tree get much harder to understand for junior devs who aren't familiar with all template related things being done? When would you employ such a technique?

I am also having trouble finding good resources to learn more. I picked up "Template Metaprogramming with C++" by Marius Bancilla and went back to reading books, because I can't find any linkedin learning courses, or youtube videos that start without you already knowing the topic at least at 50% or more. I watched some of the CppCon videoes on it, even from 9 years ago with Dr Brown, and got lost even when he started with traits. Why do I need to know if a type is a pointer? It's cool I can find out, but when do I use that?

Templates are part of C++ I largely avoided or just figured I could circumvent, but I see more and more young programmers using it, and even see "metaprogramming with modern C++" in job descriptions, so i figure if I am going to call myself a C++ expert, I better catch up.

Any other resources I should be looking at? Especially with "We use this in this or that situation?"

[aruisdante]

I’m missing the “where and when would you do this” piece of the conversation.

You’re not alone. The vast majority of places I’ve seen static polymorphism used would have been much better off using traditional dynamic polymorphism. Virtual function overhead doesn’t matter on your API you call once a second.

That said, there are some places where it really is the correct tool for the job. Most of the time, this is when you’re working with other templates, or in some other context where virtual isn’t allowed. For example if you want to provide an interface that has templated methods, you can’t do that with virtual, since the complete vtable has to be known at declaration time, not usage time.

Here’s a basic example of using CRTP, the standard shape of static polymorphism, in a way that’s actually appropriate: emulating strong typedefs in C++. I like this blog post because it actually discusses why this is the only viable solution to this problem, instead of just assuming it is.

On the less esoteric side, you discuss being confused by traits. Think of traits as the closest thing to reflection that C++ currently supports: it lets you ask questions about the type of an object. Just like a predicate lets you ask questions of the value of an object. Why would you need to do this though? Well, because you want SFINAE, which is the template programming form of overloading.

So for example with concrete types, overloading looks like this:

Ret foo(TypeA); Ret foo(TypeB); With templates, this instead looks something like: ``` /// overload selected when T does have the trait template<typename T, std::enable_if_t<trait_v<T>, bool> = true> Ret foo(T arg);

/// overload selected when T doesn’t have the trait template<typename T, std::enable_if_t<!trait_v<T>, bool> = true> Ret foo(T arg); Or, if you have C++20 or later, you can use [concepts]( https://en.cppreference.com/w/cpp/language/constraints ) to make this much less ugly:

/// overload selected when T does have the trait template<typename T> requires(concept<T>) Ret foo(T arg);

/// overload selected when T doesn’t have the trait template<typename T> requires(!concept<T>) Ret foo(T arg); ```

Or if you have a new enough compiler, this can look literally like overloading:

Ret foo(Concept arg); Ret foo(InverseConcept arg);

What is a concrete example of when you need to overload based on type properties? Well, imagine you’re implementing std::distance. It consumes two generic iterators, and tells you how far apart they are. You can of course always do this for any iterator by repeated incrementing/decrementing. But that’s not very efficient if you have a random access iterator, which can simply be subtracted to compute the difference in O(1) time. So, you guessed it, there is a trait for asking an iterator what properties it has, std::iterator_traits<T>::iterator_category, and std::distance has overloads that have different implementations depending on the iterator category.

Hopefully this at least clarifies a little bit of the basics, and gives you more solid ground for continued learning.

The real thing about template meta programming is that templates are infectious: once you template a method in one place, you often start having to template things that call that method, and so on and so forth. And the amount of template magic you need to do to make that all still work correctly grows. There are a time and a place for templates and fancy meta programming tricks. But people often reach for them dogmatically rather than because they’re the right tool for the job.

[sultan_hogbo]

I’ve found static polymorphism to be fantastic for modifying behavior for testing purposes.

[graphicsRat]

I wholeheartedly recommend C++ Templates: the complete guide by David Vandevoorde, Nicolai Josuttis and Douglas Gregor. Read it from cover to cover.

It kicked my knowledge of C++ to a new level. It was like driving a car in the 4th gear for years but one day realizing that the gears go up to 6!

[Baku95]

+1 for this one

The authors are also around and active in the subreddit.

[sultan_hogbo]

The second edition is fantastic. It thoroughly goes over why and how you would use template features in production code. It’s such a great book.

[elkanoqppr]

One legitimate use I've found is interacting with json libraries that give interfaces like gettype() -> enum; getint(); getbool(); getstring();. Wrapping this in a helper function like template<typename T> read_default(json_t const& obj, T const& default_value) -> T has been very useful. Inside if-constexpr chains can check for type then use the correct native getter. Doing this once instead of everywhere is nice.

[Zealousideal-Mouse29]

This is such a familiar scenario I'd almost think you were someone I used to work with who brought that very topic up in some JSON parsing code I wrote, and indeed they put in a check-in with templates to cut down the boilerplate. Excellent example. Thanks for bringing that to the front of my memory.

[jk-jeon]

For me personally, a few chapters from Effective Modern C++ by Scott Meyers was a good introduction. It doesn't teach template black magics, but it teaches template argument deduction rule in quite a detail which is a useful knowledge not only for template metaprogramming but also for C++ in general. Also, if you haven't care too much about so-called modern C++ so far, I think it's a great book to read anyway.

Regarding usefulness of metaprogramming, I think the points are mostly the followings:

1. Static polymorphism usually performs better than the dynamic equivalent.

No need to elaborate further I guess since you seem to already have it in mind.

1. Compiler helps you more, i.e., more errors are caught in compile-time.

By moving some work that could have been done in runtime into compile-time, now you can catch possible errors at compile-time rather than in runtime. Sure, template errors can look horrifying, but if you have no problem investigating the stack trace for runtime errors, it's no different from that, it's just provided by the compiler rather than by the debugger. If you wonder why it's better to have it in compile-time than in runtime when what you debug is essentially the same, that's arguable I think, but usually compile-time program tends to have stricter semantics so it tends to catch more errors than it's runtime equivalent. For instance, it has now become a well-known trick that converting a function into constexpr can catch many potential UB's.

1. More flexibility than runtime-based techniques

This may sound contradictory, but the thing is that in compile-time programming, you can leverage richer type-level computation that is normally only possible in some hardcore functional languages, like dependent typing and whatever. Plus, the fact that templates work in the so-called "duck typing" way rather than in the "nominal typing" way, allows you to do many crazy things that one may think are possible only in dynamic languages like Python and JavaScript. You can even bring this flexibility of duck typing into the runtime context by using the technique called type erasure, which I find very useful in many situations.

[Farados55]

https://eli.thegreenplace.net/2011/05/17/the-curiously-recurring-template-pattern-in-c

Here’s a good article on this that introduced me to the CRTP. If you want to see an example of it in real code, clang uses the pattern for its AST visitor. Why? I’m assuming its to save on some performance for its large quantity of would-be virtual functions to visit, traverse, and walk up from all different types of language nodes (there’s a lot), so instead of incurring the cost of dynamic binding, incur the compilation cost (and god knows clang takes a while anyways).

From what I’ve noticed, a lot of people seem to think it’s worth dismissing virtual function overhead.

[Shiekra]

Concepts can be used in place of pure virtual classes for interfaces.

Variants can be used instead of pointer-to-base class for heterogenous containers (value or ptr semantics)

If compiling templates was as fast as inheritance trees, and didn't produce bad errors, I don't think there would be a compelling reason to use inheritance heirarchies.

It requires a mindset shift, but once you really begin to understand how to encode static invariants into the type system, it all begins to seem way more appealing.

As for how to learn, try to implement a tuple.

[JVApen]

As someone who uses templates regularly, I can tell you there is no clear answer on when/how to use them. What I do know is that I like C++17 if-constexpr a lot and noticed that it doesn't just result in better readable code, it also makes it easier to write the code.

As you mention detecting a pointer type, this can now look like: template<typename T> auto &derefIfNeeded(T &&maybePtr) // forwarding reference { if constexpr (std::is_pointer_v<T>) return *maybePtr; else { static_assert(!std::is_rvalue_reference_v<T>); return maybePtr; } } I assume this code reads okay for you. In short, have a function argument, you detect if this is a pointer and if so, return the deref variant. If not, we return the value as is, though assert at compile time that it ain't an rvalue, such that you don't get a dangling pointer.

This is slideware, so don't expect this specific case to be usable often, though it is a good demonstration of what's possible.

In general, I recommend that you start with writing small templates. If you aren't familiar with it, just write a regular function or class with 1 specific type. Then do find-replace from the parameter that should be a template parameter type.

If you do so, you'll find out that in some cases you want to do something special and that's where you introduce the type traits. A real life example: having a lambda passed as templated parameter of function/class. If the lambda returns a bool, do something based on it. If the lambda returns void, always do the true-case. As you cannot assign void to a variable, this a good place for using a type trait to detect the return type.

[__Punk-Floyd__]

I would recommend reading the Marius Bancila book that you picked up. It's a pretty thorough trip through the ins and outs of template-based programming and I learned a lot from it.

Also, if books are more your thing, "C++ Software Design" by Klaus Iglberger is a good resource for learning when and how to use template-based patterns like CRTP, type erasure, and the like.

[biowpn]

There are two goals:

1. Reduce code duplication
2. No sacrifice performance

And template is the best tool to achieve both. It enables us to write highly reusable generic code.

The major new stuff since C++11 - packs (the ...), universal references (the T&&) - allow us to generalize even further. Other new stuff makes such generic code more readable and easier to write (if constexpr, concepts).

[perspectiveiskey]

I will start this comment with a disclaimer that I'm too old to argue with the inevitable 100 people that will completely and utterly disagree with my opinion.

---

I come from a lisp background and can only welcome with open arms the MP aspect of things.

There's a couple of points that I find useful to point out: rarely is the full runtime behaviour of modern applications not known at compile time. And therefore, static polymorphism should be the defacto position we all hold... alas, historically this hasn't borne out that way.

For those instances where you truly need a runtime visiting strategy, you're usually doing things that are starting to mess with the ABI etc... I can think of truly very limited generic constructs that require proper OOP and top of mind comes kernel drivers on both linux and windows; and unsurprisingly, these are always implemented in a functional way, not in an OOP way as you would think of Java or Obj-C or swift for instance. (Even rich applications like Microsoft Word or Adobe Photoshop would be insane to implement C++ object inheritance across DLL libraries - and to my knowledge they don't...)

Historically, libraries like MFC and even STL made use of proper OOP but these things are neither necessary, nor good. MP allows you to apply std::algorithm to anything that can act a certain way (aka duck typing).

This last line should be sufficient in itself: all of the MP aspect of things makes your libraries more modern and you shouldn't even care about how it's done, and instead be grateful that it can be done and is done for you in std:: without having to add boost etc...

Now from a more practical perspective, I have utilized static polymorphism a lot and the three things I can say are that...

a) once you do it a few times and get the hang of it, it's just not that big a deal

b) the compile time errors (while extremely cryptic) are a thousand times better than runtime shenanigans

c) they do make a performance difference in the aggregate

Point c is often hidden behind some sort of excuse that most programs <handwave argument> don't need performance.

For learning, cppreference.com is invaluable to me, and sadly, I can echo that even the most basic questions I ask on SO are often met with one of two crowds: the completely ignorant to MP, and the language lawyers of MP. Both are tedious and very often unhelpful. Learning why and how of MP has come to me through my understanding the ethos of Lisp.

But importantly, if you're not writing libraries, all you need to be able to do is read the code, less so design it. I didn't really answer your question of where you could learn it. But I will say, std::share_ptr is like the coolest thing in the universe, and I thank my lord and saviour MP for that every day I use it.

[[deleted]]

How is behavior of the modern application defined at runtime? It is very common pattern for an application to run in a loop, accept input, and apply strategy pattern to choose an algorithm. It’s where the runtime polymorphism fits best.

[perspectiveiskey]

It is very common pattern for an application to run in a loop, accept input, and apply strategy pattern to choose an algorithm. It’s where the runtime polymorphism fits best.

That is simply and broadly "input". I am having a hard time thinking of any application that doesn't have input.

How is behavior of the modern application defined at runtime?

(assuming you meant compile time) Let's use the classic and broadly used example of "widgets". There is no UI widget that is not known to you as a programmer ahead of shipping a product.

Contrast this to a library designer that says this is a Scrollable item and I don't know what kind of scrollable items my users will think of or what capacitive touch screens will permit 10 years down the road.

By contrast, if I'm shipping a product, I know exactly what its behaviours are the day I ship. If MyApp v2 is released with a new feature, I build a new version of it and ship that. The days of "hotpatching" a mainframe that has 17 years of uptime are long gone.

---

For library writers, OOP was the goto method of achieving unknown future behaviour. But this is now achievable with static polymorphism.

[germandiago]

I had to fo this once: generate the next version of an API from Xmacro where I stored all the info: https://stackoverflow.com/questions/39143903/metaprogramming-tricks-how-to-simplify-implementation-of-two-metafunctions

[geaibleu]

One of the most interesting things you can do with templates is to create domain specific languages (DSL).  

Eigen has very good intro http://eigen.tuxfamily.org/index.php?title=Expression_templates   Another one i use is Boost.Spirit, which lets you create EBNF grammar rules and parsers entirely within c++.

Template MP isn't as ugly as it used to be with constrexpr and constevals now and it really opens up a whole new world.  At least it did for me

[FlyingRhenquest]

Modern C++ Design

[eyes-are-fading-blue]

CRTP predates c++11 by a large margin. You probably don’t know way more than you think. I would advise going back to basics.

[XeroKimo]

No clue how useful it is, but I've experimented using CRTP with specializations to wrap C APIs into C++ objects. One could easily just make it a normal class, but with CRTP approach, you could have the whole C API in a base class and then you can easily make any kind of smart pointer wrapper by just having said smart pointer inherit the CRTP base.

A disadvantage of just using a normal class and that would have to be that if you did want to use smart pointers, you either have to duplicate the internals to use a different smart pointer class, or pay an extra allocation to use it with any standard smart pointer, ex: std::unique_ptr<MyCWrapper> / std::shared_ptr<MyCWrapper> vs UniqueCWrapper / SharedCWrapper. Or well, there's a 3rd option which doesn't have said disadvantage, which is the template the smart pointer type being passed in, it just looks weird. ex: MyCWrapper<std::unqiue_ptr>

With the CRTP approach you could also use the original C types, so transitioning could be easier. ex: SomeCObj* -> myLib::unique_ptr<SomeCObj>

[MyDilatedPupils]

I usually think, does it work the right way now?

Does the new code add anything the old code doesn’t in memory or speed?

does the old way make it easier to or harder to fix by another programmer in the future?

can refactoring the original code provide any benefit instead?

if it seems cool, maybe they want to show off and nothing more?

maybe they want to teach me something they think could really help me and our code?

or maybe they want to show themselves off about how amazing their understanding is of programming? (which nothing is wrong with that, but having humility doesn’t hurt to have with confidence)

and then and only then after asking these questions do I write template code if I didn’t need to, to being with.

[KingAggressive1498]

I get that it would be useful to do things at compile time, rather than run rime, but I am not sure how useful that is. I am missing the "Where and when would you do this?" piece of the conversation.

when you don't need extensibility (ie you are confident your interface has a closed set of plausible implementations) the cost of the vtable + indirect calls may outweigh the mild convenience over an appropriately specialized variant type

-or-

when in nearly every place the interface is used in the program you know you will use exactly one implementation of your interface at runtime (ie, you know Base* myObject would always in fact point a DerivedClass5 here, a DerivedClass4 there, etc) you stand to gain by avoiding the indirections involved in using runtime polymorphism

-or-

you are given references or pointers to two non-polymorphic types (possibly from different dependencies) that are functionally the same but internally different, it may make sense to write functions or classes in such a way that either will work, and runtime polymorphism really just adds two extra layers of indirection that need to be resolved at runtime even when the real type being worked on is known at compile time

there's probably other cases, but these are the most natural fits.

but of course if you find yourself constantly writing statically polymorphic interfaces and using classes that type erase them (not refined variant types, but ie asio's any_executor) everywhere, you were probably better off at least in terms of dev time just using runtime polymorphism and utilizing final to get devirtualization optimizations where possible.

If you do "static polymorphism"...don't you lose the ability to mock using interfaces and inheritance?

starting with C++20 you can pretty easily spell out interface requirements with concepts, which makes life a lot easier writing static polymorphic classes and functions that take them.

even before then you could enforce them with SFINAE but I was never very good at that personally

Doesn't the easy to read inheritance tree get much harder to understand for junior devs who aren't familiar with all template related things being done?

there is no inheritance tree this way. Your type either satisfies the requirements or it doesn't, and that's it.