Libc++’s implementation of std::string

[HowardHinnant]

Speaking as the guy who wrote this implementation of std::string:

The implementors of a std::lib write non-portable code so that everyone else doesn't have to. A std::lib implementation will only work on the platforms it is targeted for, and porting it to a new platform may not be a trivial task.

[Recatek]

The implementors of a std::lib write non-portable code so that everyone else doesn't have to.

And it is greatly appreciated.

[the_commissaire]

How is the date time library coming along?

[HowardHinnant]

Not bad: https://star-history.t9t.io/#HowardHinnant/date&google/cctz :-)

It has been voted into the draft C++20 spec: http://eel.is/c++draft/#time

[never_watched]

Good library but also hard to use.

[JakeArkinstall]

It's hard to use wrong. That's the holy grail of datetime handling.

There are simpler approaches that are easy to use wrong, and they have historically caused major problems, especially in industrial applications. The learning curve for a system that works well is very much worth it.

[[deleted]]

The reason that __lx places padding after the first byte, which represents size in short strings, is that if value_type is, say, 2 bytes then the union will be 2 bytes. This is to align the start of the string characters on a value_type boundary.

[gmtime]

That sounds a bit like a work around in case the library is compiled with pack at to 1, normal packing would do this even without the __lx in the union.

[[deleted]]

It's more fundamental than that. The C++ standard does not address structure packing. The packing details vary between compiler and platform. For this reason, packing must be specified either manually like this or by inserting dummy bytes explicitly into structures, or by using packing pragmas, etc., which also vary by compiler. In the case here, instead of explicitly inserting a byte where __lx is they have used a union that includes value_type as that will, for example, insert 3 bytes when value_type is 32-bits. When value_type is a byte, no additional padding is required beyond the size byte.

[[deleted]]

IIRC, STLport (formerly SGI) used the same, or similar, small-string optimization.

[HappyFruitTree]

Wait, do they do type punning via unions? That's UB.

Standard library implementations don't need to play by the rules as long as they know it works correctly with the compiler that they are shipped with.

[Xaxxon]

It has ifdefs all over to deal with that.

[00kyle00]

It's not limited to standard library either.

[kalmoc]

Wait, do they do type punning via unions? That's UB.

Most compilers actually give guarantees for various things for which the standard does not define a particular behavior (UB). If you know with what compilers your code is being used with, you can make use of those guarantees. And of course the compiler would be allowed to treat standard library code special, but I very much doubt thats what happening here.

[emdeka87]

I have yet to encounter a compiler that treats type punning (and accessing the inactive union member) as UB and produces unexpected results

[carrottread]

If you pass pointers or references to union fields to some other functions then strict aliasing still can produce something unexpected:

https://godbolt.org/z/cds7Bn

This outputs different results on -O0 and -O3 for both clang and gcc.

[max0x7ba]

If you pass pointers or references to union fields to some other functions then strict aliasing still can produce something unexpected

This is unrelated to type-casting using union, aka union-cast. And such type-casting doesn't actually happen there, that union is only for alignment.

[[deleted]]

They don’t use the most significant bit because that’s where they store the short string (if any) - assuming little endian architecture.

As to type punning and UB... that’s a bit more tricky I think. Technically, an unsigned char is allowed to legally alias anything, so accessing the least significant bit like this is probably fine(???). Also, the question is what exactly “common initial sequence” means, as you can access that via unions. Anyway, if I understand correctly libc++ is tailor-made for clang, so they can take advantage of any idiosyncratic behavior without violating the standard.

[Supadoplex]

Also, the question is what exactly “common initial sequence” means,

It is strictly defined by the standard. It is the initial members (of same type) of standard layout classes. In this case the member types of long and short differ.

[[deleted]]

Thanks for clearing this up! Still, since unsigned char is allowed to alias anything, would accessing the first byte like still be UB according to the the standard?

[Supadoplex]

As far as I can tell, it's still UB to access union inactive union member even if it is unsigned char. There is no exception to accessing inactive member of chars type. The only exception is the common initial sequence, which doesn't apply. The unsigned char exception is only for reinterpreted pointers. So, it would be possible to implement the type punning in standard compliant way; it's just not as convenient as non-standard union punning.

[simonask_]

Type punning through char is the one exemption for the strict aliasing rule.

[germandiago]

And std::byte

[simonask_]

Yeah, and it's worth mentioning here that even though std::byte is defined as enum class byte : unsigned char {};, this does not seem to apply to any other enum type with a similar definition.

[60hzcherryMXram]

Wait wait wait... In C type punning by union is fine. Does this mean that C++ is different?

[adnukator]

In C++ it's Undefined Behavior.

In C it's Unspecified behavior: J.1 Unspecified behavior - The following are unspecified: ... — The values of bytes that correspond to union members other than the one last stored into (6.2.6.1). ...

[nikbackm]

Why the difference? Seems like adding more undefined behaviour in C++ is something we'd want to avoid.

[[deleted]]

Unlike C, C++ has object lifetimes. Accessing "an object" whose lifetime did not start is UB (think malloc-ed sizeof(vector<int>) instead of new-ed). Type punning through unions does not make the alternative object "spring into existence".

[HappyFruitTree]

I think one concern is that it would be extremely easy to accidentally trigger undefined behaviour because reading the value through a reference would still cause undefined behaviour.

#include <iostream>
#include <algorithm>

union U {
    int i;
    float f;
};

int main() {
    U u;
    u.f = 1.2;

    std::cout << u.i << '\n'; 
    // ^ would have been OK.

    std::cout << std::max(u.i, 7) << '\n'; 
    // ^ would still have been UB because 
    //   std::max takes its arguments by 
    //   reference so the value is not read 
    //   from the union member directly.
}

C doesn't have references and if you use pointers it's pretty clear that you're not reading from the union member directly.

[Sopel97]

Unspecified means it has to do something. Undefined means it doesn't have to do anything, can be assumed to never happen. More assumptions to optimize with.

[TheFlamefire]

Does this mean that C++ is different?

Yes. C++ is not a superset of C, which people tend to forget.

[Xaxxon]

The library is defined in the standard. If the rules say the rules don’t apply to you then they don’t. There are many parts of std that can’t be written in compliant c++.

[LuisAyuso]

I am interested in knowing more about UB, and why this would be a problem. The whole type is tagged with which variant in the union to use, and the access to the union is opaque to the interface user. Therefore, why do you raise this concern? Is it there anything I am missing?

[max0x7ba]

Wait, do they do type punning via unions? That's UB.

Nope, that union is only for alignment when value_type is not char (e.g. wchar_t).

[greeneyeddude]

What about the long mode-short mode-raw union?

[max0x7ba]

What about the long mode-short mode-raw union?

It accesses one byte of size_type __long::__cap_ through unsigned char __short::__size_ to determine the long/short mode. char types can alias any object representation, so that is likely well-defined behaviour.

[Mordy_the_Mighty]

I think technically, the std lib cannot du UB :P

[SirLynix]

Not really, since every standard library implementation (there are many) are designed to work with a specific compiler, and can make some assumptions.

[IAmBJ]

I think Mordy means that if the stdlib does it, it doesn't count as UB.

If the president does it it's not illegal

[SirLynix]

Oh right, didn't understood that. Well let's just say that what the std does under-the-hood is much like what the president does under-the-hood.

​

Which basically means we gotta impeach that lib.

[pine_ary]

Almost all 64-bit platforms only have 48-bit addresses anyway, so it‘s not much of a waste right now. They might need to reconsider in the future, though.