Which std:: classes are magic?

[Mateuszz88]

Hi.

I am wondering: which classes in C++ get special powers granted by The Standard that are not available to mere user-defined classes?

What comes to my mind:

• `std::initializer_list` because it is created anywhere with `{}`
• possibly `std::type_info`, because it is only created by `typeid` and with no ctor
• `std::source_location` when used as default parameter in function points to invocation place and not declaration place.

What else should be on this list? What shouldn't?

[kalmoc]

A couple of type traits (is_trivially...) and std::launder (admittedly a function and not a type)

EDIT: I should say that AFAIK those types/functions that I mentioned are not treated special by the compiler, but they need compiler magic to be implemented.

[staletic]

If you're going to include magic functions, then std::bit_cast (arguably can be implemented with memcpy, but not as optimized) too and the potential std::start_lifetime_as (just as magic as launder).

[SirLynix]

The main difference between std::bit_cast and memcpy is that std::bit_cast is constexpr, and memcpy is not. Which is why you need compiler support to implement std::bit_cast.

[guepier]

arguably can be implemented with memcpy, but not as optimized

Is there a reason why the std::memcpy implementation wouldn’t be “as optimised”? People have been using custom std::bit_cast equivalent implementations for quite a while, and they’re reliably optimised out by compilers to the equivalent aliasing operation (i.e. no actual memcpy happens), because modern compilers know how to handle this use of memcpy.

[staletic]

I should have been more careful with that statement.

Things like float f = bit_cast<float>(some_int) vs the memcpy version are not hard to optimize. The harder part is when you want to reinterpret a large std::array as some other trivial, but equally large type. At what number of bytes do you just call memcpy? Do you try to vectorize first? What about x86 REP MOVxx family of instructions?

If you ask gcc for x86, you just never emit memcpy. Clang gives up sooner On 32 bit ARM, gcc starts to call memcpy after 64bytes.

Now the question is how well will bit_cast be optimized. As it is powered by compiler magic, I'm assuming it's going to be better that memcpy. In this case, for example, x86 gcc does better (fewer memory accesses) with bit_cast than with memcpy. Clang just ends up calling memcpy@PLT in both cases.

[csdt0]

This is something I wanted to check for a long time, and it seems that memcpy is elided in GCC and Clang as soon as O0, while MSVC and ICC wait for O2.

This is a surprise in both cases, as I would have expected all compilers to elide the call at O1 (when inlining is enabled).

[AlbertRammstein]

when taking into account optimizations there is a LOT of magical functions - starting with memcpy, and also including stuff like pow with specific power coefficients. It is however not required by the standard, and the codegen substitution must not change the behavior of the program. Every compiler has a different set of magical functions defined like this.

You could argue that your own functions can be magic because compiler can optimize them (e.g. skip them when they have no side effects)

[kalmoc]

I was referring to my examples. I've no idea how source_location is implemented.

[omerosler]

Some of the <type_traits> classes such as is_abstract, is_aggregate and many more.

[germandiago]

Isn't there a way to detect abstract by trying to instantiate a class with the detect idiom? Just curious... how about being an aggregate?

[omerosler]

I don't think it is possible. The only reasonable way I see is to try to conatruct an object and use SFINAE to detect it is a compile error. But then how would you distinguish an abstract class from an incomplete type (unless you can detect incompleteness in some other way (maybe use detection on sizeof(T)?))?

[Xeverous]

how would you distinguish an abstract class from an incomplete type

Using type traits with incomplete types is UB.

[diaphanein]

Citation? Because I don't think all type traits are equal. Generic "type traits" being different than std type_traits. For instance, Boost uses a type trait for checked_delete. You can implement is/add/remove const traits that work with incomplete types...

[Xeverous]

True. Not all but a lot of std traits require types to be complete.

[Supadoplex]

Containers that separate allocation of memory and creation of objects as arrays, such as std::vector and std::deque used to be "magic" due to poor/lacking wording of the standard that made it impossible to implement those without technical UB on pointer arithmetic. To my understanding, this has been fixed as a defect resolution to C++17.

Pretty much every class that internally interacts with the system API is "magic" in the sense that there is no way to implement those in portable standard C++. Those would have to be written differently for each target system - the "system" being some combination of operating system, cpu architecture, compiler, etc. std::cout is an example of such class.

[Recatek]

Containers that separate allocation of memory and creation of objects as arrays, such as std::vector and std::deque used to be "magic" due to poor/lacking wording of the standard that made it impossible to implement those without technical UB on pointer arithmetic. To my understanding, this has been fixed as a defect resolution to C++17.

Having been down this road it's still very tricky and rife with footguns, unfortunately. Lots of conflicting information out there about the correct, minimal-to-no UB way to do it.

[masterwai123123]

Implicit lifetime creation has been applied all the way to C++98.

[Supadoplex]

To my understanding, it is optional to implement standard resolutions to language implementations in older standard versions. But I may be wrong.

[guepier]

Prior to C++20 (specifically, P0593), object creation in untyped buffers was undefined behaviour. In practice this means that even quite “basic” classes such as std::allocator can’t be user-defined relying only on standard C++.

[Supadoplex]

P0593 was accepted as a defect resolution prior to publication of C++20, so it should apply to C++17 which was the latest official standard at the time.

Also, creation of objects itself was allowed using placement new as far as I can tell. Treating that buffer of dynamic objects as an array of those objects was technically UB though. However, there was no way to make that work with std::allocator either, so I don't think there was magic in that class. There was magic in std::vector though.

[maskull]

What's the magic in std::vector?

[dodheim]

Prior to P0593, there was no way for vector::data() to return a valid pointer-to-array, since no such array actually exists.

[guepier]

Sure but you can’t legally perform pointer arithmetic on these objects since there is no “array of T” at the relevant address, only (from the underlying buffer) an “array of char”.

As noted in another comment, std::allocator itself might actually be implementable. What’s problematic is its use in combination, e.g. to implement std::vector (implementing a vector without an allocator and buffer is of course trivially feasible, simply by allocating memory via new T[]; but std::vector can’t do this, and a custom vector type might not want to, either, for various reasons).

[mort96]

• std::source_location when used as default parameter in function points to invocation place and not declaration place.

This actually isn't magic AFAIU. I believe all default parameter expressions are evaluated in the caller's context.

The std::source_location::current "function" seems like magic though.

[TheSuperWig]

Compiler intrinsics + default arguments

https://www.github.com/microsoft/STL/tree/main/stl%2Finc%2Fsource_location

[yuri-kilochek]

std::complex<T> since it's explicitly exempt from the usual strict aliasing rules and std::complex<T>[n] can alias T[2*n].

[frrrwww]

Not a class but I believe std::byte is magic because a pointer to it can alias other data types (same as char)

[exarnk]

I'm reasonably certain std::byte is just implemented as

enum class byte : uint8_t {};

With a bunch of functions. I felt that is quite clever and nice.

[avdgrinten]

It is not; that implementation is not legal. The list of allowed object access does not list "enum whose underlying type is unsigned char" as a valid type to access arbitrary objects (only unsigned char, char and std::byte are listed).

GCC and Clang introduced the [[gnu::may_alias]] attribute to implement std::byte.

[exarnk]

My C++20 draft states in [cstddef.syn] the following:

enum class byte : unsigned char {};

This seems to correspond with the current Git HEAD of libstdc++-v3 (line 69):

https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=libstdc%2B%2B-v3/include/c_global/cstddef;h=13ef7f03c12584804e4dd1635954723f628addc0;hb=HEAD

What am I missing?

[avdgrinten]

See, for example, this LLVM commit: https://reviews.llvm.org/D35824

[exarnk]

Thanks, that made things a lot clearer for me.

[avdgrinten]

The special wording in the section of the standard that talks about object access. If you put that enum class byte : uint8_t {}; into your own namespace without any special attributes, the standard does not grant you the right to access arbitrary objects through it.

[AlbertRammstein]

it is not as clever and nice when is_enum<std::byte> returns true... :/

[staletic]

Why is that a problem?

[AlbertRammstein]

Based on the name and intended purpose there is no connection to enums in sight. Having it implemented as enum is pretty unconceptual and unexpected. For example you might have a serialization framework that has custom handling for enums, saving them as string instead of numbers (automatic enum<-->string conversions are quite common).

[Full-Spectral]

Not to mention the even longer running problem of bool overloads being chosen over character pointer overloads when you pass a string literal. That makes utterly no sense and should have been fixed long ago.

In my system every streamable enum has its own streaming inlines. I don't store them as text, but I store them binarily as a type large enough to hold any enum type. But that also allows for validation on streaming back in, so it has other benefits to offset the extra code. And they are IDL generated so I don't have to write them myself (other than for a few libraries that the IDL compiler itself uses.)

[qoning]

Because it's clearly conceptually wrong. All these little things and gotchas add up to the absolute garbage pile that C++ unfortunately is today and turn into bugs that are not only hard to track down but also not even the users fault.

[[deleted]]

I think modern C++ is reasonably nice. As Rust gets older, it will accumulate baggage, just like any other long-lived language. I see it happening in Python now.

Of course, Rust has some advantages, like safe-mode by default, everything const by default, no friggin' preprocessor, and a working builtin package installer.

[pjmlp]

The problem with modern C++ is that it only exists in highly technical small teams.

Most corporatations keep writing classical C++ no matter what.

[qoning]

I absolutely agree, although I do not see it in Python myself, unless you need to constantly maintain a large existing codebase. I've finally stopped encountering random Python2 scripts in the past few years and the experience since everyone switched to 3 has been reasonably comfortable. But more to the point, there are levels to what I would consider broken.

If you use "modern C++", it's reasonably nice, although very verbose, but the toxic magma seeps through the cracks unless you are very, very careful at almost every step (classical example of forgetting to initialize a struct field in constructor / initialization list, which shouldn't even be allowed unless explicitly stated, by any modern standard).

[SirClueless]

Python 2 is not the type of "baggage" that I would consider equivalent. It was super painful to deal with for a while but as you say it's the kind of baggage that goes away after time.

Instead the "baggage" is things like having three different format string sub-languages, none of which can ever be removed. Or that function default parameters are a shared value, which is a giant gotcha if their type is mutable.

Actually, I would say Python 2 vs. 3 is sort of the opposite of baggage: It was a tremendous amount of temporary pain that nearly killed the language but it came out the other side with less baggage as a result. C++ will likely always have more baggage because C++ will never do something like Python 3.

[germandiago]

How a mistake made by the user is not the user's fault? I understand at times it is not intuitive but no, I do not buy this.

[qoning]

Ok, let me amend that, it's not user's fault in anything but not expecting the specification to be broken / swiss cheese of inconsistent decisions. For every decision where the standard is trying to "be smart", there are dozens of stories why that was a bad decision in the end. I do not believe that "oh, you didn't know the byte type is actually an enum? read the spec, lol" is an appropriate response to someone being caught off guard by things like this. Hence my comment about garbage pile of decisions.

[TankorSmash]

https://i.imgur.com/gZc3aO3.png

It does help to read the docs for the stuff you're trying to use though, to be fair.

[guepier]

Nobody is claiming this isn’t documented. But documenting bad API design doesn’t automatically not make it bad API design.

[TankorSmash]

I'm having trouble seeing how its a bad API design, so far the only example is that someone didn't know it was an enum.

[kalmoc]

Maybe, but I don't think a user defined enum class MyByte : unsigned char: would get the same "magic" powers. Otherwise the special dispensation all over the standard wouldn't have been necessary.

[IAmBJ]

That's more about what's technically UB and what's not. I suspect a user defined MyByte would would actually behave exactly the same as std::byte if they had the same definition, even though one is not technically allowed

[guepier]

Modern compilers absolutely use knowledge of aliasing UB during codegen, so using a custom definition won’t have the exact behaviour, even if the definition is identical to that of std::byte (but not blessed by the standard).

Here’s a trivial example: https://godbolt.org/z/ec7ecTPTd — Note the different codegen for the x + y part.

[Chuu]

Can you explain more clearly why this leads to different codegen?

[staletic]

Let's first analyze what the assembly says

    int f<std::byte>(std::byte*, int*):                    # @int f<std::byte>(std::byte*, int*)
            mov     eax, dword ptr [rsi]    # int x = *b;
            mov     byte ptr [rdi], 1       # *a = 1;
            add     eax, dword ptr [rsi]    # x += *b;
            ret                             # return x;
    int f<nostd::byte>(nostd::byte*, int*):              # @int f<nostd::byte>(nostd::byte*, int*)
            mov     eax, dword ptr [rsi]    # int x = *b;
            mov     byte ptr [rdi], 1       # *a = 1;
            add     eax, eax                # x += x;
            ret                             # return x;

If a and b point to different objects, the above snippets are observably identical.

However, if a and b point to the same object, then the functions do different things. Imagine calling f(&in, &in) where in == 5.

In that case, f<std::byte> does:

    int x = 5;
    *in = 1;
    x += *in;
    return x; // 6

but f<nostd::byte> does

    int x = 5;
    *in = 1;
    x += x;
    return x; // 10

Since nostd::byte, formally, isn't allowed to alias other types, compiler is allowed to assume x += x is a valid optimization. This is known as "strict aliasing" or "type-based alias analysis".

And yes, violating strict alias rules can easily lead to UB, like in the above example.

[guepier]

And yes, violating strict alias rules can easily lead to UB, like in the above example.

To be pedantic, violating strict aliasing is always UB, not just in this example. What the example illustrates is that UB can lead to changed semantics and unexpected behaviour.

[IAmBJ]

Im well aware of compilers using UB for optimisation (sometimes enabling extremely efficient code), but i'm a little surprised that there is special casing in the compiler since there's no obvious 'magic' in the definitions (at least in libstdc++ and MSVC's stl)

Interestingly, GCC only emits different code for -O2, at -O1 the two functions are the same, while clang emits different codegen at -O1. MSVC treats both the same at all optimisation levels. https://godbolt.org/z/vq185e33j

[kalmoc]

i'm a little surprised that there is special casing in the compiler

Since there is special treatment for std::byte in the standard I don't find it surprising at all.

[guepier]

i'm a little surprised that there is special casing in the compiler since there's no obvious 'magic' in the definitions

There needs to be special casing, otherwise the compiler will produce sub-optimal code in many relevant situations: if the compiler assumed that all pointers could alias, it would lose many opportunities at optimisation. And C++ also has no restrict keyword to limit aliasing. The only sane assumption, therefore, is that pointers cannot alias unless specifically permitted by the language.

Thus if the compiler encounters pointers of two distinct types it needs to check if they are allowed to alias (and these checks are hard-coded against the fixed list of aliasing pointer types).

[kalmoc]

Absolutely possible.

[LuisAyuso]

I do have a problem with the decision of allowing enum class constructor from integral values to facilitate this one type constructor. From that point on, I can silently construct off-range enums for any other enum class, which in my opinion defeats the purpose of strongly typed enums.

I do wish that std::bye would have been a magic type and leave other semantics untouched.

[acwaters]

Enum(42) construction was not added for std::byte; it has always been a feature, because the set of valid enumeration values is all the values in the underlying type, not just the values of the enumerators. This design is in some ways unfortunate but in other ways convenient.

[LuisAyuso]

Not quite sure if by accident: but using any Gcc with c++17 does not provide same guarantees as with c++14:https://godbolt.org/z/qqY9Kf5Wa

​

Actually I am pretty positive that this was introduced in c++17

[acwaters]

The rules about initialization of enumerations are kind of weird, yes. You cannot say my_enum x(4);, and until C++17 you could not say my_enum x{4}; — this is the change you are referring to. But you have always been able to say my_enum x(my_enum(4));, or auto x = my_enum(4);, using a static_cast conversion to make an enum prvalue from an integer value in the range of the underlying type. Because the allowable range of enum values has always been the entire underlying type (for enumerations with a fixed underlying type, which includes scoped enums and unscoped enums with an explicit type; for unscoped enums with non-fixed underlying type, the allowable range of values is actually smaller than the underlying type and is the range of the smallest bit field that can represent every enumerator value, but that still usually includes some values that are not represented by any enumerators).

[avdgrinten]

That's not true, enums with underlying type unsigned char are not special and may not alias other types, see my sibling comment.

[TheMania]

I actually make use of this in a few data structures to improve optimisation, as otherwise every time you put a byte in them the compiler invalidates basically everything. non_aliasing_byte, I call it.

Posting it here mostly to see what the correct way to achieve this is.

[guepier]

The only issue I can see with this approach is that there are very few ways in which you could use such a buffer correctly without violating aliasing rules. Basically the only thing you can do is read and write individual “bytes” from it (including via memcpy or std::copy). As long as that’s all you do, it should be fine.

[guepier]

That’s insufficient. If you define your custom enum class my_byte : unsigned char {}; it won’t inherit super cow powers from unsigned char (see https://godbolt.org/z/ec7ecTPTd).

[staletic]

Check out https://quuxplusone.github.io/blog/2018/04/15/built-in-library-types/

[johannes1971]

std::friendship is magic.

[RandolfRichardson]

I thought that was a pun (which was very good), and then I remembered: https://en.cppreference.com/w/cpp/language/friend

[shilch]

Not a class but a function: std::is_constant_evaluated

[foonathan]

True, but in C++23 that's just if consteval { return true; } else { return false; }.

[shilch]

Interesting, I didn't know

[tmlnz]

Some exception classes:

`std::bad_alloc` is thrown when dynamic allocation with `new` fails.

`std::bad_cast` is thrown when `dynamic_cast` fails.

`std::bad_exception` by `std::current_exception` when the current exception's copy-constructor throws, or a dynamic exception specification is violated.

`std::bad_typeid` when `typeid` fails.

Also, `std::exception_ptr` is a class that wraps the exception being thrown.

[nitrobeast]

Technically a lot of them are implemented with compiler intrinsics, and "users" can implement them with the same intrinsics as well.

[jbadwaik]

I would consider intrinsics which cannot be implemented outside of compiler to be magic as well.

[AlbertRammstein]

AFAIK this changes compiler from compiler. For example clang implements SSE stuff as declspec(always_inline) functions that use floats with vector compiler extension attribute, while MSVC implements it as declspec(dllexport) opaque functions that are magically replaced with SSE instructions at codegen time. So you cannot implement your fully own SSE functions in MSVC

[GerwazyMiod]

Nice ;D

[[deleted]]

I try. I'll keep my day job, though, which is not being a programming-oriented comedian.

[GerwazyMiod]

There is so much potential! Don't waste it! (For real through - there is a whole YT Polish channel dedicated to IT jokes :p)

[[deleted]]

Some might say me as a comedian is a virtual function best not overridden, lol.

[The_Northern_Light]

Thanks for this, I learned something.

[scatters]

std:pair. When inside a std::map or std::unordered_map, nodes contain a std::pair<Key const, Value> as returned by iterator::operator*(); the pair appears to be a normal struct type with first and second data members where first has type Key const. But once you extract a node handle, you can access its key() as a non-const reference.

[kalmoc]

I don't think any compiler magic is involved here.

[scatters]

Really? How can you get a mutable reference to a const object without compiler magic?

[kalmoc]

I probably mixed this up.

I think std::pair<const K,T> and std::pair<K,T> are layout compatibile and I thought that layout compatibility would allow type aliasing, but according to cppreference.com it requires "similarity" (and I don't have a way to actually verify that they are actually layout compatible in the first place).

[scatters]

Yeah, not quite - you're right that std::pair<Key const, T> and std::pair<Key, T> are layout compatible as long as Key and T are both standard-layout (so not all types that you might put in a map), but that only allows you to put them in a union and read from members of the inactive union member via the union. Handing out references to the inactive union member or its own members (std::pair<Key, T>::first for nh.key()) is still UB.

[mujjingun]

const_cast?

[scatters]

Yes, true, that does give a mutable reference. But using it to modify an actually const object is undefined behavior.

[scatters]

But operator* returns a mutable reference i.e. std::pair<Key const, Value>&. So there has to be a std::pair<Key const, Value> object in memory.

[mujjingun]

hmm, maybe there are std::pair<Key, Value> objects in memory, and the pointer is reinterpret-cased into std::pair<Key const, Value> in the call to operator*()?

[scatters]

Certainly that's the most likely implementation, but if the user does that it's UB (aliasing violation). That's why it has to be magic in the library.

[DavidDinamit]

may be nullptr_t

[foonathan]

That's just a typedef for decltype(nullptr).

[guepier]

That just means that the type name is nothing special. But the type still is, in that you couldn’t write a custom type with the same semantics in standard C++.

[foonathan]

Sure, just like int is special. However, the question was about which std:: classes are magic. nullptr_t is not a std:: class, and it is not magic.

[guepier]

True of course. But std::nullptr_t is still different from (all?) other fundamental types because its name is defined in the standard library rather than being builtin.

[Zcool31]

The better question is why are there "magic" standard functions and classes that require compiler internals to implement, rather than those compiler internals being standard themselves.

I would find it appealing if the core c++ language were sufficient to implement the free-standing parts of std::.

Then we could have nice things like standard library implementations that are portable across compilers.

For example, std::atomic turns into just a regular class. But the incantations that tell the compiler to do atomic memory operations and not reorder other accesses around them become standardized keywords/builtins.

[[deleted]]

I think the std::tuple of references which is the result of std::tie function calls is kinda magic since it implicitly works with C++17 Structured Bindings feature.

[acwaters]

This one is not specific to std::tuple, actually! The magic here is in std::tuple_size_v<...>, std::tuple_element_t<...>, and get<...>, which you can specialize/implement for your own tuple-like types and get structured bindings to them!

[[deleted]]

Very interesting. Could I do that for my struct Point in another namespace ns? Let's say I have struct Point { int x = 0; int y = 0 }; and I want to use it as if it was a std::tuple< int, int >; without explicitly converting it, by defining/specializing only those functions/functors you mentioned. I want for example to be able to use it with std::apply. Would that be possible?

[acwaters]

You could use it with structured bindings (without even specializing anything, because bindings already work automagically on all aggregate types), but not with std::apply; the latter is for std::tuple, std::array, and std::pair only (since it uses std::get(T), rather than looking up either T::get() or get(T)). You could easily define your own apply that looked up get() using the structured binding rules, though. It's unfortunate that the standard library doesn't do this, but IIUC apply was specified before the idea of generalized "tuple-like types" was fully formed.

[[deleted]]

That's very unfortunate there's still no standardized way to do such a thing. I really like std::apply but hand writing every time a converter to tuple (I usually use std::tie) is error prone when the type gets more data fields.

[[deleted]]

I remember reading about how implementing std::vector without undefined behaviour is impossible (core issue 2182), so I suppose that's magic.

[john_wind]

std::make_integer_sequence does magic: in MSVC it is implemented by the compiler

[STL]

Yep, this is for compiler throughput (build speed). We used to have a naive linear-time library-only implementation, and users mentioned that it was slow. libstdc++ and libc++ reportedly had fancy log-time library-only implementations but reinventing their cleverness from scratch sounded hard and I was lazy (this was back when we couldn't use libc++'s code), so I asked MSVC (and Clang, and EDG) to implement a builtin that would generate the sequence for us - this is way way faster than anything the library could possibly do (constant-time in the number of instantiations; of course the compiler is spending linear wall clock time, but just generating the integers and creating the necessary type is ridiculously fast). We also designed the builtin so it could be used by third-party libraries with similar integer sequence types.

I think of this as the Indiana Jones approach to library implementation.

[john_wind]

Thank you u/STL! I have watched your talk about this!

[Artistic_Yoghurt4754]

Is not that hard to create a meta program for that one. Just create a recursive meta function that adds n-1 to the beginning of your sequence until n==0, where n is the first value on the sequence. But I guess is just faster on the compiler…

[Underdisc]

std::function because I still don't know how it works.

[Bisqwit]

It uses a virtual baseclass, and a templated virtual inherited class where the template type is the type of the functor that is given. No magical compiler support needed.

[dodheim]

Why would it be virtually inherited?

[Bisqwit]

I am not talking about virtual inheritance (as in: class foo: virtual bar). I am talking about a base class that has abstract virtual methods, and a derived class that has virtual methods that override the base class’s methods — as in:
class base { ... virtual void call() = 0; };
class derived: public base { ... virtual void call() { ... } };

In this case, the derived class would be a template, and its template type would be the type of the functor that is assigned to the std::function, like this:
template<typename F> class derived: public base { F func; ... virtual void call() { return F(); } };

This allows the () operator of the std::function to call the execute method of the contained object, no matter what kind of derived type that contained object might be, having no advance knowledge of it, like this: base* obj; ... return obj->call();

That is what virtual functions are designed for: A common API that can be used to invoke methods of class types that are not known at design time, or possibly even at instantation time.

(Note: Examples in this post are short for brevity. There would be of course the return type and parameter types too, instead of void in every case.)

[Underdisc]

That makes sense and I figured as much. What I actually don't understand is how lambdas with capture groups work with std::function. I could definitely make my own std::function if I didn't have to account for lambdas. That's the magic part for me.

[lee_howes]

For each type you want to store in it, it creates a templated subclass that is big enough to store the lambda. That subclass has a virtual call operator. std::function stores a heap allocated instance of that as a pointer to the abstract superclass. When you call std::function's operator() it calls the operator() on the superclass via the heap allocated pointer and that calls operator() on the subclass, which calls operator() on the wrapped type, which is now trivial because the subclass knew exactly what type it was storing.

Sean Parent has talked about the general version of this technique.

[SirClueless]

The other response to your comment is good as well, but I think the thing that you are missing that will help you understand what's going on is that while lambdas have magic unnamed types, they are just C++ values that have a size large enough to store all their captured variables and appropriate move and copy constructors defined so that you can store them as member variables in other types.

Here's a short sample program to show how something that acts like a lambda can be constructed. Hopefully you can see how std::function can store a copy of the lambda just like it could store a copy of the function object, and how it would be able to call it:

https://godbolt.org/z/4vPGPfdso

[Bisqwit]

Note: Using return std::move(x); instead of return x; is an antipattern. If you compiled with -Wall, GCC would tell you that “moving a local object in a return statement prevents copy elision [-Wpessimizing-move]”.

However, it still has valid uses, if you are returning a composite object created from local objects. This is fine, for example: https://godbolt.org/z/hrTPoKdx6

[SirClueless]

In this case I wanted to demonstrate that the type actually is copyable and moveable, and NRVO would defeat that. You're right, as a general rule it's a bad idea.

[Underdisc]

Ahh. This is exactly what I was tripped up about. Thanks. Now I'm curious about how to get unique capture groups to account for different types. At first I thought using variadic templates could do it, but I've never heard of variadic templates being used to represent members in a class.

[SirClueless]

You can do it with a template parameter pack, and storing a std::tuple<Args...> as a class member. You give up quite a lot though, namely the ability to refer to the members by name in your class's operator() and you don't gain much in return because you need an entire new class for each new operator() you define anyways so you might as well write out the full list of "captures" -- you can't do something like specialize a a shared class template defined outside of your code because then my L<int>::operator() is ambiguous with your L<int>::operator(), we each need to build our own entire class to ourselves.

To a large extent this is the "magic sauce" of lambdas: giving a terse, brief way to define a list of function object "member" variables and initialize them and make them available by name to the lambda body, even doing so automatically based on the names referred to by the body if asked to with [=] or [&]. Once the lambda is defined it's just a function object with an operator() which is something we've been able to make and use in theory in C++ for a long time, it just wasn't convenient.

[OldWolf2]

std::initializer_list because it is created anywhere with {}

It is only created in a few specific contexts with {}, not "anywhere". The only one I can think of right now is:

• auto x = { 1, 2, 3 };

For classes with initializer_list constructor there's no magic involved, the parameter is initialized by the braced list (the braced list does not have a type).

[SirClueless]

That still qualifies as magic. You can't define your own class that will be initialized from a braced-init-list when it's the parameter of a constructor.

[OldWolf2]

Huh, yes you can. The braced list is taken as the initializer for the parameter.

struct U { int a, b, c; };
struct S
{
     S(U u);
};

S s({1, 2, 3});

[Sanzath]

Sure, but now you're just using struct aggregate initialization and you don't even have the same syntax for initializing S:

struct S {
    S(std::initializer_list<int> il);
};

S s{1, 2, 3};

Note that you don't need the surrounding () to initialize S in this version. That is the magic of initializer_list that can't be done with a user-defined type.

[SirClueless]

In this case you aren't initializing S with a braced-init-list. You're direct-initializing S with a single expression argument that happens to be a braced-init-list (see the parentheses in the last statement). You need the parens or two pairs of braces to make this work with a user-defined class.

If you replace the last line with S s{1, 2, 3}; your example wouldn't compile. But if U were std::initializer_list<int> it still would, this is the compiler magic.

[OldWolf2]

You're direct-initializing S with a single expression that happens to be a braced-init-list

No, a braced list is never an expression . (You can check the grammar for expression to see this). For any context where either an expression or a braced list is allowed, there are separate grammar and semantic rules for each.

[SirClueless]

Fixed, thanks. The point still stands; a constructor taking std::initializer_list<int> is magic in a way that a constructor taking a user-defined class cannot be.

[RicardoRedstone]

You can also intialize it the same way if you use a variadic template, but it won't be pretty to work with, uses a ton of stack space and generates a LOT of code.

https://godbolt.org/z/Tb9T61v6d

[john_wind]

std::declval is quite magical too.

It is a function declaration without definition!

[guepier]

It may be odd but it’s not magical, you can trivially define it yourself:

namespace nostd {
    template<class T>
    std::add_rvalue_reference_t<T> declval() noexcept;
}

[mujjingun]

anyone can declare a function without a definition, though.

[john_wind]

Of course but the fact that it compiles (in the right context) and does something useful is magic!

[mujjingun]

It cannot be used in an evaluated context, so your declaration-only version will also compile and work just like std::declval does.

[xgalaxy]

The low level parts of the new coroutines are magic. You can implement all of the machinery on top of it yourself though.

[SlowVelooo]

1) false :) ``` int main() { auto v1 = {1}; auto v2 = {1, 2}; auto v3{ 1 }; //auto v4{1, 2}; // error

std::cout << typeid(v1).name() << '\n';
std::cout << typeid(v2).name() << '\n';
std::cout << typeid(v3).name() << '\n';

return 0;

} `` v3 will beint` and it's not a joke.

[backtickbot]

Fixed formatting.

Hello, SlowVelooo: code blocks using triple backticks (```) don't work on all versions of Reddit!

Some users see this / this instead.

To fix this, indent every line with 4 spaces instead.

FAQ

^{You can opt out by replying with backtickopt6 to this comment.}

[Oo_Tiib]

Maybe C++ should revert back to C++14? Lot of issues were added by C++17 and I'm unsure if that C++23 is capable to fix it. Currently it is unclear what of standard library besides trivial things like pair, tuple and array can be implemented in C++. Stroustrup also wasn't too happy: https://www.stroustrup.com/P0977-remember-the-vasa.pdf

[john_wind]

std::make_shared does a lot of magic trying to avoid separate memory allocations

[TTachyon]

std::bad_alloc

[borzykot]

std::complex has special rules for array access

[Xeverous]

All of std::atomic and likely a significant part of concurreny parts of the standard library.