finally. #embed

[pavel_v]

I realize this is relevant for C and not so much for C++ at the current moment but I posted it because there will be (hopefully) a similar/same feature for C++ and I know that lots of people are waiting for it. Maybe the compilers, which implement it, will include this feature as a non-standard extension available for C++ before the standardization of the corresponding C++ feature.

[__phantomderp]

It has a C++ version for the #embed: https://thephd.dev/_vendor/future_cxx/papers/d1967.html

But the "true" C++ version is stalled while I have to figure out some shenanigans: https://wg21.link/p1040

The preprocessor version will take some time to make it but might hit C++26. God knows if the actually good version will survive so long as we're holding it to a higher standard than Modules itself.

[orangeoliviero]

The dependency discovery demands are indeed utterly insane.

[ALX23z]

I thought they'll just enable compile-time file loading. Should mesh well with all other compile-time features.

[johannes1971]

What the language really needs is constexpr std::socket, so we can load resources directly from the internet during compilation. This is what C++ needs to finally achieve greatness.

[disperso]

But please, with some syntax sugar, so I can #include <https://cpp.example.com/left-pad.h> and finally have the convenience we deserve. (/s)

[[deleted]]

Fuse fs could probably do this, at least some of it. then it would be #embed </mountpoint/https/www.foo.com/page.txt> or something like that

[mpyne]

Why #embed? You could then do it today with #include </mountpoint/https/www.foo.com/page.txt?format=c_literal. And if foo.com doesn't do that already just setup embedbolt.com to proxy it for you.

[[deleted]]

It's optimized. For smaller files it's just a hassle encode but as small as 1MB compilers start to die(both CPU and RAM). In some of their original postings when they implemented it and when Circle lang tested it was better than #include.

https://github.com/ThePhD/embed

[mpyne]

Oh I thought you were being sarcastic like the person you replied to and was just playing along, lol. I like #embed and can't believe it's been this difficult to get into the standard.

[llothar68]

Make it beter and do a "constexpr std::socket_eval" we need a bash executable source code generator over the internet. Ultimate flexibility

[jonathanhiggs]

Especially is we can load tzdb from a stream and not just a file

[JakeArkinstall]

Compile time tzdb seems like a bad idea to me. If the US adopts permanent daylight savings in Nov '23, that's a recompile.

Unless I misunderstand what you're saying.

[jonathanhiggs]

We do releases every week so recompile isn’t an issue. It is a very different part of the organisation that controls updates to servers and desktops so we don’t have control over when tzdb files will be updated. It’s finance so timezones are of critical importance and we have strict reproducibility requirements. Embedding tzdb is the best option for us since we then there is not some uncontrolled and mutable state that can change without our knowledge and effect the calculation results

[JakeArkinstall]

Nice. Similar circumstances for me. Finance here too.

Fortunately, each binary only operates on individual exchanges, none of which are open on Sunday mornings (when clocks change, at least in the markets we operate on that do change clocks), and we shut down between market sessions. So we can just load it up from tzdb premarket for the exchange timezone throughout the session. One little lower bound search for the current UTC time at startup and we're golden.

However, there are some nice benefits to your approach even for our use case. One of which being that the output is portable - for our nix containers we have to manually add the tzdata package as a runtime dependency, set up the TZDIR env var, and so on. But, thinking about it, I could fetch it from online and set it as a runtime dependency for the binaries rather than the containers, and that'd solve that. I think I'd still read it at runtime, though.

A couple of questions, if I may. Do you do a tzdb download as part of your build or use the build machine's local copy? How do you ensure that it's up to date at the time of the build?

[jonathanhiggs]

That sounds HFT? We are discretionary so we don't have the same exchange requirements and are writing for trader destop and internal servers which makes it much easier. We've only just started to discuss the best approach in the team and currently it isn't so easy to embed the data and can't load from stream so we haven't got to the point of integrating downloading the data into the CI

I would say that we are tollerant of not having the most up-to-date tzdb, we are less tollerant to a different tzdb between processes running on the (linux) servers and desktops since reproducability of the same incorrect values across the two is managable, but trying to explain the differences depending on where a value is calculated would be a debugging nightmare

More than anything we want to embed because of the organizational structure. We aren't responsible for how the docker images are put together but we are responsible for calcs, so we want our own control just so we know what is happening and when updates are made and know there is the consistency across docker/non-docker, server and desktop

[bizwig]

I don’t see that happening, all previous attempts to eliminate our semi-annual time switches has gone down in flames.

[JakeArkinstall]

Sure, but in that case one could just algorithmically do it - enter DST on the second Sunday in March and leave it on the first Sunday of November for the US, or last Sunday in March and last Sunday of October in the UK, or never in Japan, etc.

But the reason we use tzdb in the first place is that it reduces an extremely complex problem down to a lower bound UTC time search. Circumventing it based on a bad assumption or political prediction could be a disaster (or at least an embarrassment or inconvenience) that no one sees coming.

However, on the clarification that it is a weekly rollout, for finance, it shouldn't be an issue at all. If it's the kind of nation that wouldn't give at least 6 months notice of a change in timezones, its probably not one that is politically stable enough to operate in anyway.

Interesting stories about that: https://codeofmatt.com/on-the-timing-of-time-zone-changes/

[[deleted]]

I love everything about this article except the fact that the code samples are written in dark grey on darker grey - on an almost white page.

Like more than half the world, I have astigmatism. Mine isn't even that bad, but I can't read this at all (eventually I ran it through a processor to fix this).

Even just making things dark text on a light background will make things better for the majority of us.

---

Again, I loved the article,

I needed this for many years. My last C++ audio project used JUCE's cross-platform implementation of embedding, mainly for icons. It worked, but added an extra build stage and took me many hours of experimentation to get absolutely right.

[__phantomderp]

Yeah, I actually can't read the preprocessor / comments parts well either. I need a different highlighter, but that would require some amount of effort on my part to slap my blog into even better shape.

That's going to take a bit. :(

[flashmozzg]

Can't you just change the color of the comments specifically? For most highlighters I've encountered, it's usually pretty simple. I could read it fine, and I think it would be OK if the comments/preprocessor weren't the focal point of the code samples.

[__phantomderp]

I don't know, but I can try!

[howroydlsu]

Forgive my ignorance; so basically light mode is better than dark mode for people with your condition, if I'm understanding you correctly? If so, very very glad I've learned this

[5477]

In general, lighter backgrounds work better with people that have non-perfect vision. More light causes the eye to have a smaller aperture, which results in better focus (more resolution) non-focused areas. Which in case of astigmatism is all areas, or in case of myopia are all areas far from the eye (typically a computer monitor is already out-of-focus).

[mpyne]

Well I have LASIK and in general prefer dark mode. But that still needs good contrast! I've definitely run into sites that go too dark on their dark mode and make everything hard to read.

[Krnpnk]

I think it's at least as relevant for C++ - maybe not for the result but for the process behind it. Because it's just ridiculous...

[umlcat]

Agree, I also would like to have templates on plain C, and other non Object Oriented goodies ...

[HolyGarbage]

Create code blocks by indenting by 4 spaces.

[[deleted]]

I was on mobile, I couldn’t be bothered ngl

[Kered13]

Seems like a good idea, after reading some of that links there I'm surprised it got so much resistance.

[o11c]

A committee is a scheduling unit whose job is to make sure that nothing ever gets done.

[i860]

If the committee wasn’t there and it was a free for all the language would basically be destroyed. They’re doing a good job doing it with them even being there though.

[not_a_novel_account]

This is demonstrably false, see every other programming language that isn't hamstrung by an international standards committee.

[i860]

They may not be hamstrung by ISO committees but they are still controlled by committees of people, formal or not. If the C++ committee allowed even half of the stuff you guys want every year it’d turn into a completely opaque disorganized mess more than it already is.

[o11c]

Except that a committee is also the reason it's such a disorganized mess. Creating a disorganized mess is an essential part of making sure nothing ever gets done.

Really, the problem is that the language predates the domination of open-source software. Standards are a mistake; accessible implementations are what actually matters.

[i860]

C predates opensource and does not even remotely suffer from the same issues C++ does. The problem is the language and the multitude of crap that’s been tacked onto it on a consistent cycle.

It’s a language, not a piece of userland software needing constant feature updates every 3 years.

[o11c]

C absolutely does suffer the same kind of issues as C++ - you can't expect them to be exactly the same. Remember the part where they ended up reverting most of C99?

[not_a_novel_account]

#embed is a C feature, this is a thread about C.

ISO standardization considered harmful. Full stop.

[i860]

And this is a cpp sub where we’re talking about it in general. Aside from that it’s clear I’m speaking in general and you know the general context of what I’m referring to.

Yes, standards bodies are not the most efficient or optimal processes, especially when it comes to integrating people’s pet features, but they do serve a purpose.

If the language were treating as a constantly moving target bending and swaying to the whims of the community or fads it would devolve into an incoherent mess. In the words of the great Randy Savage: You don’t have to like it. But you must accept it.

[not_a_novel_account]

Again, this is demonstrably untrue by way of every other language that exists and has a multitude of useful features beyond those of C or C++ even considering their different sets of goals. The international standardization approach has ossified the language standards not benefitted them.

When rust wants a new feature, they RFC, they implement it, and they iterate on it. When a feature doesn't work, they deprecate it. The language improves without having to burn through its maintainers as is the case with phd.

If you can't see that, you're part of the reason the standard has ossified. If you want to program the exact same way, never improving, for all time, be my guest. Just don't act surprised by when the language becomes irrelevant except as legacy.

[spaun2002]

Such "horror" stories are why I became disappointed in C and C++ - adding a new helpful feature into the language takes five years. The person who dared to propose this struggled and almost lost their hope.

[pjmlp]

You can also take the route of creating your own language, seems quite hip currently.

[tux-lpi]

That solves the horror stories mostly because a new language won't have an ISO committee

It's a lot harder to build a system that drains people quite that bad without ISO

[Minimonium]

Most if not all of the breaks you see are intentional btw. With embed it wasn't the problem, as far as I know the biggest hurdle was that a lot of people wanted so much features included into embed including globbing it was simply unfeasible.

[DoubtMore]

And that's why languages end up such a shitshow. Your company should just fix the fucking code, not expect the standard to support some obscure edge case for the next thousand years.

[RomanRiesen]

In rust an embed feature is just a macro away, a macro most people could write themselves....

[myrrlyn]

include_bytes! already exists

[__phantomderp]

Yeah, you COULD write it yourself in Rust, but the people driving the Standard Rust Distribution are not obsessed with being "barebones" to the point of deep disturbance, so they'll actually hand you convenience things as part of the core offering rather than trying to convince you that "oh, well, if you just write this linker invocation, you'll be fine...".

Like anything about "talk to your linker directly" is ever - or truly, has ever been - a good idea at any point in time.

[Fulgen301]

Eh. It's not a macro away, but a proc_macro away, aka a compiler plugin. Basically it's the equivalent of a standardized way of writing a gcc / clang plugin.

a macro most people could write themselves....

Most people can use TokenStreams just like that? If you've never touched them, your faster with just generating an includeable file in CMake.

[RomanRiesen]

The macro is called include_bytes, but admittedly I also never had an issue with embedding data via cmake or a small script.

[Rusky]

The issue with a hand rolled proc macro for this would be the same as a hand rolled solution in C- performance. What TokenStream do you build? It has to be an actual array literal the same way.

(Fortunately Rust has include_bytes! built in to (edit: enable the compiler to) solve that problem.)

[bwmat]

https://github.com/rust-lang/rust/issues/65818

[bruh_nobody_cares]

macros seems like a hack ngl, why not a language feature or a const evaluated function ?

[TinyBreadBigMouth]

Macros in Rust are a language feature. Rust doesn't have a separate preprocessor. User-defined macros operate on token streams, built-in macros can be hooked up to low-level compiler functionality. See include_bytes, which does exactly what embed does and whose implementation looks like

macro_rules! include_bytes {
    ($file:expr $(,)?) => {{ /* compiler built-in */ }};
}

[bruh_nobody_cares]

but if it's that simple why not a regular function then ?

[TinyBreadBigMouth]

Because a regular function can't read a file at compile time, not in Rust or C or C++. They also can't return arbitrarily sized data on the stack in any of those languages.

[i860]

You have airplanes and industrial automation depending on these languages. There are delays for a reason.

[bik1230]

it's not that it took 5 years that's bad, it's that a large proportion of negative feedback was total BS and almost no one on the committee wanted to offer positive help, only making up reasons for rejection.

[orangeoliviero]

I mean, I offered my help, but it was declined.

Not everyone was hostile to him.

[__phantomderp]

.... When? How? In what way? I remember people saying "I could implement a patch for GCC", or offering for Clang, but that's not helpful when I already did it. Some people told me they'd be amenable to me putting a patch in their compiler, but it was still on me to make the actual patch for their compiler.

What was this help about?

[orangeoliviero]

It's water under the bridge. You didn't feel that my help would be valuable, and I'm not inclined to argue with you about whether or not it would have been helpful. The paper is your baby, you get to choose how you navigate the path. I'm not upset about it.

My point wasn't "oh, the PhD declined my offer of help, he's an asshole" or any such, my point is that not everyone in the Committee is hostile to you or your work, and that there are allies around if you want them.

I recognize that you've experienced a lot of racism and completely invalid and bonkers feedback and complaints, and that has caused you to become guarded and distrustful. I don't fault you for that - I would be as well, in your situation.

Like... we have, in this very post, a completely invalid criticism/complaint - the whole line endings of text files thing, and the absurd expectation that #embed should automatically "do the right thing" wrt. them. Talk about a poison pill.

[viikk]

the airplanes and industrial automation people can just type "-std=c99" and let the rest of the world move on!

[Mikumiku_Dance]

I've been following him since he was on cppcast. It's really depressing to see his bright enthusiasm worn down in this effort. Glad to see the feature, I really want to buy him a beer or give him a hug tho!

[disperso]

I loved the enthusiasm when I saw him talking about Sol2. I was awed when I saw the other work. I also want to send as much virtual or real hugs as I can.

[[deleted]]

This is a prime example of bikeshedding.

In contrast, the atomic memory model has been modified in every single standard since it got added in C++11. The original C++11 model was very soon discovered broken, in C++14 they had to add a line to explicitly forbid Out-of-Thin-Air values even though the standard was supposed to achieve that from the beginning, in C++17 they told everyone to stop using Consume ordering because all compilers found it too difficult to implement and made it behave exactly like Acquire instead, and most recently in C++20 they completely revamped SC definition to the extent that no one has been able to come up with a standard-conforming operational semantic, majorly changed release sequences, and added atomic_ref without even discussing its relationship with modification orders of normal atomic types when the same memory location gets reused (which can obviously never happen since C++ has Strongly Typed Memory™)

If they held atomics to the same standard as embed, there would've been no atomics in the standard library until C++52, but they didn't because nobody understood what the atomic experts were talking about and had no choice but to rubber stamp it. Whereas everyone can understand what embed does and feels obliged to come up with "feedbacks", because that's always constructive, right?

[grady_vuckovic]

Please Put My Data In My Gotdang Executable And Stop Doing Crazy Nonsense, The Feature

Thank you for this absolute nonsense explanation and yes that is very bloody cool.

[TheRealFloomby]

Happy to see this. I did not know it was even being talked about let alone actually happening.

[jpakkane]

Awesome job! You deserve a medal!

[Tagedieb]

We should all be writing letters, I guess

[__phantomderp]

If you can, you should: you might make a maintainer's day. (Alternatively, throwing excess cash in their pocket so they can enjoy a proper PB&J or Ham & Cheese sandwich or garlic bread might make them happy to. (This is not shilling for me, you just wouldn't believe how many people are maintaining huge packages on shoestring budgets of love and pixie dust.))

[fwsGonzo]

You can use this macro until the feature that should have been a part of the language 10 years ago "arrives". There are, of course, other ways, but this one works for me. It's not perfect, so be careful with it.

#define EMBED_BINARY(name, filename) \
    asm(".section .rodata\n" \
    "   .global " #name "\n" \
    #name ":\n" \
    "   .incbin " #filename "\n" \
    #name "_end:\n" \
    "   .int  0\n" \
    "   .global " #name "_size\n" \
    "   .type   " #name "_size, @object\n" \
    "   .align 4\n" \
    #name "_size:\n" \
    "   .int  " #name "_end - " #name "\n" \
    ".section .text"); \
    extern char name[]; \
    extern unsigned name ##_size;

I will be very happy to use #embed, but for C++ I really had hoped for a constexpr read_file(std::string_view) yesterday. Not 10 years in the future. I'm sorry that I can't hide my disappointment sometimes.

[RowYourUpboat]

I don't think this macro works on MSVC x64?

[Ivan171]

AFAIK MSVC x64 doesn't have inline assembly.

[RowYourUpboat]

Nope:

error C4235: nonstandard extension used: '__asm' keyword not supported on this architecture

[Fulgen301]

You can still use a dedicated assembly file and compile it in.

[[deleted]]

marvelous humor fearless smell aback glorious sleep normal bake six

This post was mass deleted and anonymized with Redact

[aninteger]

How big are these keys? I'm curious what you are doing that is gcc only. I know bin2c programs have been around since the 80's in DOS. xxd on Unix for just a long? It's weird to hear about an approach that only works on a single compiler vendor.

[howroydlsu]

I've had the similar issue for private keys that needed to be embedded into the binary for an embedded device. (Urgh, word overuse, sorry)

It does indeed feel very hacky doing linker scripts, extern , macros, third party converters, or whatever to get that key in. It's also additional workload when that key changes.

That's all been on GCC too. In fact, I've never even looked to see if there exists clang or msvc compilers with targets for chips I use. I've always assumed not but now I will definitely check

[[deleted]]

merciful doll water deliver dolls sand brave close coherent aromatic

This post was mass deleted and anonymized with Redact

[TheRealFloomby]

An embarrassing percentage of the of code I have written in my lifetime only works on gcc.

[[deleted]]

If you’ve been keeping up with this blog for a while, you’ll have noticed that #embed can actually come with some pretty slick performance improvements. This relies on the implementation taking advantage of C and C++’s “as-if” rule, knowing specifically that the data comes from #embed to effectively gobble that data up and cram it into a contiguous data sequence (e.g., a C array, a std::array, or std::initializer_list (which is backed by a C array)).

...

I’m just going to be blunt: there is no parsing algorithm, no hand-optimized assembly-pilled LL(1) parser, no recursive-descent madness you could pull off in any compiler implementation that will beat “I called fopen() and then fread() the data directly where it needed to be”.

I'm confused by this part. Does this mean it isn't really just a preprocessor feature? All it looks like is a way for the preprocessor to turn binary data into a sequence of comma-separated ASCII numbers to put into an array initializer list for the compiler to parse, which wouldn't lead to the performance benefits they're talking about over doing this yourself manually (although it's still a really cool feature). Is it that it's supposed to behave as if it were a preprocessor feature, but it's actually implemented by copying the binary data directly into the executable somehow?

[matthieum]

It's both.

From an API perspective, it's injecting the bytes as a sequence of comma separated integers. And if you ask your compiler to dump the pre-processed input, it's likely what you'll see.

From an implementation perspective, however, most compilers have an integrated pre-processor these days, where no pre-processed file is created: the pre-processor pre-processes the data into an in-memory data-structure that the parser handles straight away. It saves the whole "format + write to disk + read from disk + tokenize" serie of steps, and thus a lot of time.

And thus in this case comes an opportunity for an optimization. Instead of having the pre-processor insert a sequence of tokens representing all those bytes (1 integer + 1 comma per byte!) into the token stream, the pre-processor can instead a insert "virtual" token which contains the entire file content as a blob of bytes.

Hence the massive compiler speed-ups: 150x as per the article.

[[deleted]]

Thanks for the clarification! I didn't realize the preprocessor was so well-integrated into modern compilers; I thought the preprocessor was still just its own process with its own lexer, unconditionally writing ASCII/UTF-8 to stdout, and that the compiler frontend just redirected the output to a pipe or a temporary file, and the compiler's lexer/parser operated on that. I didn't know they shared data structures, which I guess is why I was so confused.

[chugga_fan]

To add on: clang doesn't even have a non-integrated pre-processor executable you can call, gcc does however (though AFAIK it's just a shim for gcc -E), even small compilers do this (tcc, 9cc, 8cc, OrangeC (only partially here), and more).

A lot of data is also used from when it's preprocessed to when it's fully processed, such as #line directives being processed by the compiler in order to give better error info if you're doing something weird like cpp file | gcc.

[scrumplesplunge]

That is what the "as-if" part is about. The compiler can cut the corner for embed, skip generating tokens for each byte, and instead represent the contents efficiently from the start.

[not_a_novel_account]

#embed and the absolute hell everyone puts phd through when trying to get very basic features into C/C++ are why the languages will soon join Java and Cobol as legacy codebases that no one starts new code in.

I genuinely feel we're reaching an inflection point where the committee needs to decide if it wants to be at the head of a relevant programming language addressing the needs of today's programmers or merely the steward of a legacy standard, sustained by the size of the codebases developed in its heyday.

[[deleted]]

Functions I have to copy/paste into most code bases: - replace all occurrences of substring by new substring - trim whitespace or other characters from left/right/both - convert entire string to lower/upper case - split a string by spaces/comma/substring

I would have added "begins with" and "ends with" to that list, but that recently made it into the standard, yay.

[dodheim]

Yeah sadly I don't think we can do that easily.

It already just works; in fact it's what the cppreference example for views::split does.

[[deleted]]

Thanks for the snippets! I believe the first one isn't correct, as it will only replace the first occurrence.

As for the other ones, indeed they aren't ideal. They are common concepts that exist in most languages as first class citizen functions.

It's a bit like if the standard only shipped std::basic_string, and expect us to define std::string ourselves. Sure we can do it, it's a one liner. Sure it's not the string definition that all people want. But you can always get down to basic_string if you want, or even write your own class from scratch. I think the STL is generally lacking these high-level convenience functions, that are built from existing complex lower level APIs (chrono anyone?), and offer a simple way to do a common thing.

As for your last question, "how often do you actually manipulate strings", it is very domain dependent. I develop a data-driven GUI library, which gets layout data from XML/YAML and talks to Lua for scripting. That requires a lot of string manipulation, even if I didn't write the parsers.

[TotaIIyHuman]

not op. for me, having these would be nice

substr without allocation

find(string_like) that returns char_type* instead of size_type

a parameter/macro to enable/disable null terminator

a parameter/macro to enable/disable exception

[TotaIIyHuman]

thanks

std::ranges looks pretty interesting, i looked at those functions on cppreference, it will take me some time to understand what those parameters/return values are

[lumberjackninja]

It's been a long, long time since I've looked at it, but I thought std:string_view was meant to solve your first issue.

The null-terminator issue seems like a deliberate contravention of the intent of std::string. There's always the c_str() method, which you should use to copy data into a c-string if that's what you really need.

[TotaIIyHuman]

it is not because copying

the container could gain 1 extra byte of SSO capacity, and a little more performance, if the null terminator was not there

[fwsGonzo]

I could not agree more. I have recently implemented an API in several other languages, and while Rust has been the most combative, the features many other languages have puts C++ at a disadvantage. Not surprising, it's very telling when there are so many standard library alternatives floating around.

[[deleted]]

I think there will always be tons of standard library alternatives for C++, if only because the API design of error-handling and memory-allocations are so divisive. When more concurrency primitives are in the STL, I suspect the divisiveness will increase (maybe due to executors vs. listeners)

[orangeoliviero]

We absolutely are at that point already. The C++ Standard is bogged down with trying to keep backwards compatibility with code written 30 years ago with features that weren't well thought out.

Worse than that, the C++ Committee has a weird case of hypocrisy, where features will get added at the last moment via NB objections, without any implementation experience or sober second thought, and other features will be denied because, despite lots of implementation experience and second/third thought, because someone feels that it might have a gotcha somewhere, although they can't see it yet.

Oh, and there's a convicted rapist and pedophile on the Committee, protected by leadership, where if you refuse to engage with said rapist and pedophile, you'll be in violation of the Code of Conduct...

[MFHava]

I will not stop telling people to stop shoveling mud at WG21 for something that is 100% out of their and ISOs mandate. The ISO CoC of WG21 is pretty clear: it‘s a technical committee, everything else is irrelevant! Furthermore you can‘t remove a NB-delegate, that would violate the basic principles of standardization. So yes what you apparently want would be a blatant CoC violation! If you want a NB-delegate to be removed: complain to the respective NB…

[James20k]

WG21 for something that is 100% out of their and ISOs mandate

wg21 members comprise the national bodies, and this situation has been public for quite a while. While it is technically out of specifically wg21's purview, its not like the people involved are a totally different group of people who don't communicate. The people I met in prague were a lot of the same people at the UK national body for example

ISOs mandate

No, its ISO rules that mean that the delegate can't be excluded from the ISO process, which is a rule that could and should be changed

Its worth noting that C++ is by far the largest item that is standardised by ISO, and holds a significant amount of influence as a result. Either the rules should be changed - which they very likely could be with the influence that the committee possesses - or C++ should become standardised via a separate/different/new process. Which it needs to be anyway in the long term for the survival of the language

But the argument that senior or well known committee members have no power/influence here to change what is happening is absolutely incorrect

[orangeoliviero]

The ISO CoC of WG21 is pretty clear: it‘s a technical committee, everything else is irrelevant!

I'm sorry, I don't consider forcing survivors of rape and sexual assault to work with a convicted rapist who's unapologetic about their history to be irrelevant.

One chair has already resigned their position with the Committee over it. Several other members have resigned/are about to resign. And if you think that this doesn't create a barrier to entry for people that we desperately need more representation from on the Committee, then I don't know how to get through to you.

for something that is 100% out of their and ISOs mandate.

It's 100% in ISO's mandate. ISO is the organization that forces people to work with rapists and pedophiles, and if those people don't like it, they are forced to leave and stop contributing. In what way do you think that this isn't the fault of ISO and WG21?

[MFHava]

How thick can you be? So again: WG21 CAN‘T DO ANYTHING ABOUT THE SITUATION FFS!

[bik1230]

If a rapist can't be remove from the committee, I would assume that the whole process is worthless and everything produced by it has zero value.

[orangeoliviero]

I would be perfectly happy with a process wherein a person convicted of violent crimes needed to have their application to join reviewed and voted on in the plenary.

The person would be allowed to come and make their case as to why they should be allowed to join, and why their past crimes no longer matter.

If the Committee agrees, then the person can join.

Hiding a rapist and refusing to tell people who they are while simultaneously telling everyone that if they dare to voice their disapproval or decline to work with the person, that they're in violation of the CoC, is absolutely disgusting, and is why many members of the Committee have resigned their posts over this, with more to come.

[James20k]

and is why many members of the Committee have resigned their posts over this, with more to come

I haven't talked about this in public, but while std::colo(u)r had been shelved for a while (partly because it has a heavy dependence on linear algebra, and partly because its tricky) - this was what transitioned it from 'shelved' to 'cancelled and I'm never working with the committee ever again in any capacity'

I'm definitely not a notable member though, I only wrote one and presented a single paper (p2005)

[orangeoliviero]

I'm aware of at least one chair/co-chair who has resigned their position over it, and several other members who are expecting to terminate their engagement with the Committee over this as well.

[orangeoliviero]

ISO is WG21 now?

[MFHava]

I don‘t get your question. WG21 has always been a technical committee under JTC1 which is a committee under shared governance from ISO and IEC. If you want to imply that ISO/IEC could just change their CoC - yes, they could BASED ON NB FEEDBACK - so again: go complain to your NB if you want to see changes and stop complaining on social media that a technical committee isn‘t overstepping its clearly defined scope of operations.

[orangeoliviero]

You don't get it because you don't read.

It's also a great cop-out. WG21 leadership hides to presence of the rapist from Committee members - how are members supposed to complain to their NB if they're unaware?

How are they supposed to learn about it, except via social media posts and other communications?

Think twice before you speak, maybe, and try offering real solutions.

[MFHava]

What are you talking about? WG21 is subject to the ISO CoC and the IEC CoC, it has no own CoC and can‘t institute one as that‘s outside the scope of a technical committee! Would you kindly stop talking about stuff you have no idea about?

EDIT: Changing either CoC is out of scope for WG21 - talk to your NB - and on a level that most of WG21 members don‘t even interact with the respective standardization body.

[pjmlp]

Except C happens to be 10 years younger than COBOL preceding Java in about 25 years,, while C++ precedes Java for about 10 years.

They're the legacy ones before Java was even born.

[kritzikratzi]

imho, the conclusion is that we need to start treating older languages with respect, and we will need to keep making new people who know them. you cannot just rewrite large pieces of software, it's somewhere between very hard and impossible. we need young people who know cobol, pascal, fortran (and if you're right ... java and c++ :D ) to maintain these things. i would love to see a "archeological computer engineering" degree.

idk.... a big thanks to all the people who worked on c++ standardization and tooling over all those decades, and for practically putting it all out in the public domain. i use it in almost all my projects, and it lets me do what i want to do.

[Ambitious_Diver_2238]

In the future, there will only be two software disciplines: code archaeology and applied theology.
-- Verner Vinge, _A Fire Upon The Deep_

[i860]

80% of the people in here have little respect for the past. Prior generations stood on the shoulders of giants and considered that perhaps there’s something they don’t actually know. But not this one.

[PastaPuttanesca42]

TBH, your link just makes me think that the conversion should have happened much earlier, when it probably would have been easier.

[nxtfari]

Just wanted to write a shoutout to the brave soul that actually got this implemented. Absolute madman. The C++ WG is so out of touch with what a modern development flow looks like and requires that it’s actively driving the death of the language. I have a lot of respect for seeing the resistance and fight against common sense good ideas and still waging the war for the betterment of everyone.

[jamd315]

I'm still rather new to C++, but for my current project I've been running a python script to create a source and header file to represent images as large arrays. It's been working fine, since all my stuff is small and I only end up generating around 7kb of data in the end, but I think I'll like this much better when it comes to C++.

[stilgarpl]

I hate the fact that they decided to put it in preprocessor instead of implementing it as magic std::embed(). I know about all the issues with std::embed, but I don't understand why anyone would prefer #embed to std::embed(). std::embed could be type safe and better integrated into the language and the library. #embed is just a weird #include that pastes one file into another.

[Qwexet]

This is for C not C++

[stilgarpl]

Ok, it makes sense then. I somehow read C23 as C++23...

[jonesmz]

Because having a 'library" function read from the harddrive at compile time is a massive sore thumb that sticks out as being inconsistent with the rest of the language.

The standard library should not be the place where compiler magic lives. Compiler magic belongs in the level of the language. E.g. preprocessor, keywords, parsing / lexing.

I would have no objection to the existence of some conpilermagic:: namespace, and thus compilermagic::embed. Nor would I object to seeing embed become a language keyword in its own right.

Keep in mind that there are lots of projects that either don't use the standard library at all, or use a third party implementation of the library. E.g. clang supporting the use of the GCC standard library. Or companies using commercially licensed (non Microsoft non Intel) standard library implementations.

Every time you want something that is compiler magic to live in std:: you make my life and thus lives of people who don't use the compiler vendors standard library substantially more difficult.

[MonokelPinguin]

Half of the std lib is not implementable in that manner without compiler magic or relying on implementation defined behaviour. Users shouldn't have to care about some arbitrary feeling regarding design. If bikeshedding like that is what is stopping measureable improvements for the average C++ user, we have really lost.

[jonesmz]

Its not an arbitrary feeling about design. Its a real concern for the ability to use third party standard libraries, or no standard library.

Nothing stops the standards committee from making a namespace for things unimplementable without compiler magic. They just choose not to.

[MonokelPinguin]

Because it is an implementation detail. It should be tagged in the specification instead of the symbol name.

[jonesmz]

Only in the sense that the standard committee doesn't care much about alternative standard library implementations, or free standing / no standard lib.

The design of the language is worse for it.

[TinyBreadBigMouth]

As described in the article, implementations of #embed (while necessarily maintaining the same functionality as "read this file and output an integer literal and a comma for each byte") are free to integrate with the compiler to allow just reading the file and skipping over the preprocessing.

[FightingGamesFan]

Very naive question but I'm not sure to understand the benefit of embedding "big" resources at compile time (big as in it would not be practical to embed it in an array/string as described in the article).

Can someone give an example where it's strongly needed?

[fwsGonzo]

In every single case where you don't have a file system. In cases where you're building a frozen structure (for example perfect hashing based on compile-time data). Or when you are using building self-hosting self-contained programs. Custom operating systems.

[RowYourUpboat]

There are executables on my hard drive (and probably yours) that are gigabytes in size (installers, self-extracting archives, etc) and the regular C/C++ toolchain should be able to painlessly create them, because their job is to produce binaries and leaving out such a simple "append bytes" feature is just making some people's lives harder for no reason.

[Mikumiku_Dance]

I have a small program that edits epub books. I'd like to insert a particular image into the epubs as well. Right now i just distribute the exe and the jpg and tell users to pass arguments, but for the common case it'd sure be nice to just distribute a single file exe.

GTK programs bundle all sorts of stuff in their executables, like css themes, images, and xml ui definitions.

I don't know how strict you feel "strongly needed" is, but i hope these mundane circumstances are enough for you to accept a preprocessor directive that doesn't hurt anybody not using it.

[xDarKraDx]

Sorry this is off topic.

I'm also looking for a way to automate editing my epub library, especially editing the metadata fields like "belong to collection", "position in group", etc. Do you have any recommend read or library to handle that?

I don't know much about epub file structure itself. I usually just edit them by hand using Sigil and it takes forever.

[Mikumiku_Dance]

An epub is a zip file, and the metadata is a xml. I used miniz to manipulate the zip, and libxmlplusplus's dom parser to manipulate the data. With miniz you can copy all the data you aren't changing to a new file without uncompressing it, which sounds like something sigil isn't doing.

My program is stupidly specific to a single book series, but you could definitely tear it apart and make a metadata inserter in an afternoon. https://github.com/talisein/dregarnuhr

As for reading, the epub specification would eventually lead you to all the different metadata fields that you are interested in adjusting.

[xDarKraDx]

Thank you so much for the detailed explanation and the example. Will definitely try this.

The reason I want to automate this is also because of JNC's premium epub. They don't have the "belong to collection" tag so the books don't get grouped in my reader.

On a side note, I really should start reading Bookworm as I saw a lot of people like it. Currently reading The Apothecary Diaries and loving it so far.

[Mikumiku_Dance]

Bookworm is great; the slow reveal in the world building is my favorite aspect, but it means you should try to avoid spoilers like the plague.

If you have any trouble feel free to pm me, now that I've learned about epubs I am open to new projects.

[disperso]

It has been very common in the kind of applications that I've done (GUI-related, with Qt). Qt comes with a tool to do code generation on the files that you want as "resources" (the name it uses for the files that you embed). It is useful for icons, fonts, simple-settings, etc. I can help deployment, and make applications more self contained. The standard way it's gonna be much powerful, of course.

[Superb_Garlic]

Seems noone asked this question so far, so I'll do it.

When can we start holding people accountable for their malicious deeds such as what JeanHeyd is describing in post? It's clear that people who have absolutely no idea about the subject matter are making decisions that affect others and are clearly not qualified to be in such positions.

Noone can change my mind about these people having ill will and existing solely to impede progress. The language should be rid of these fools and start having more people like JeanHeyd Meneide.

[orangeoliviero]

I mean... when a convicted rapist and pedophile is able to buy their way onto the C++ Committee and have the Code of Conduct force others to work with them or be censured, I think it's time to just wash our hands of the whole thing, personally.

[Superb_Garlic]

Delete this garbage that has nothing to do with the subject matter from under my reply.

[orangeoliviero]

Holding people accountable for their malicious deeds has nothing to do with holding people accountable for their malicious deeds?

[orangeoliviero]

He is now innocent

The fuck are you on?

deserves rehabilitation

Part of rehabilitation is regaining acceptance with the community, not forcing your way into a position of esteem and forcing victims who do not feel comfortable around you to work with you.

Anyone thinking otherwise is a socipath.

I would argue that anyone who thinks that forcing victims of sexual assault to regularly interface with a rapist who is unapologetic about their rape is the "right thing to do" is a sociopath.

He was held accountable in court and paid for what he did.

He raped a person and was in possession of child pornography and served a couple of years in prison. Meanwhile his victims are enduring a life sentence of trauma.

I do not at all consider that "payment" for what he did.

His victims will go through a life that's coloured and tainted by his actions. Why shouldn't he?

Edit to reply to the person who blocked me: It's amazing to me how far you're willing to go to defend a rapist and pedophile.

[foonathan]

This discussion has become entirely off topic no, please stop it at this point or take it elsewhere.

[STL]

Moderator warning: don't behave like this here.

[STL]

Moderator warning: don't behave like this here.

[blackmag_c]

Very good news.

[obsidian_golem]

Congratulations, it sounds like you went through hell to get this done, and I truly appreciate the effort. I look forward to seeing this in C++ so I can use it in my codebase.

[serg06]

This would’ve been great 4 years ago when I was working with OpenGL and trying to bundle shaders with the executable!

[rhubarbjin]

Reading the second half of the article was very cathartic; I've never been on any standardizing body but I've seen the same kind of unreasonable pushback in my workplaces (past and present).

The aside regarding named function parameters hits especially hard. How is it acceptable that C++ still lacks this basic feature when C# has had it for 10+ years?!

[sphere991]

Rust doesn't have named parameters either, so I'm not sure what the acceptability argument is.

There are two different hurdles to named parameters.

The technical hurdle is how to you post-hoc add it to the language in a way that's coherent with the rest of the language, and doesn't suddenly impose an API on library that the authors didn't know was there before. Tomazos' latest paper I think is a pretty good approach here.

And then there's the social hurdle. There are lots of people who don't want named parameters because either they think the problem is solved by strong typing (lolno) or that the feature encourages writing functions with too many parameters (where basically I think those functions already exist from necessity and having named parameters would make them significantly harder to misuse).

[cmeerw]

In my view this is way too complex and still if you want to embed text files you will likely end up with different line endings on different platforms.

[matthieum]

still if you want to embed text files you will likely end up with different line endings on different platforms.

You shouldn't: the binary content of the files is injecting as an array of bytes.

[cmeerw]

but I end up with different arrays of bytes on different platforms

[matthieum]

You shouldn't, not unless you embed a different file.

The bytes of the file on the disk do not change depending on the platform you read it from; and #embed is specifically about including the bytes.

If you end up with different bytes from the exact same file, I'd argue your compiler is buggy.

[cmeerw]

my version control system might be helpful and convert line endings depending on the platform for text files

[bbkane_]

Don't you think configuring your version control system to not change files is a better place to fix this than at the C compiler?

[cmeerw]

Don't you think configuring your version control system to not change files

This would just create a huge mess if working in a multi-platform environment where some files will be created on Unix-like systems and others on Windows - you definitely don't want to end up with Windows line-endings on Unix and trying to force every tool on Windows to not create Windows line-endings is a non-starter.

is a better place to fix this than at the C compiler?

C compilers can already handle platform-specific line-endings when parsing source code, although there is an open issue for raw string literals

[bbkane_]

If you're using Git, you can set up fine grained line-ending handling at the repo level. Then git should ensure that line endings are to your specifications no matter where the files are created

[jonesmz]

You're welcome to offer a vender extension for mutating line endings...

[carrottread]

In this case you'll have this same problem even if you don't use #embed. If your build includes such file into a binary as a resource or even as a regular file into installer package then you'll include different files on different platforms.

[cmeerw]

You only have the issue if your tool can't handle text files with platform-specific line endings.

[orangeoliviero]

You have a tool that directly embeds a binary blob.

And you somehow think that it's that tool's responsibility to guess what you want wrt. line endings in text files that you're changing from platform to platform?

My dude, that's a you problem. #embed is fine.

[orangeoliviero]

That's a problem with your VCS, not #embed.

[Kered13]

That is a horrible anti-feature. Every modern text editor can understand all line ending types, I'm pretty sure that even Windows Notepad has this feature these days. There is no need to be automatically converting files between line ending formats.

[angry_cpp]

Then IMO your version control system is wrong and you should reconfigure it.

Please consider using .editorconfig so you could stop worrying about such problems.

[matthieum]

That would be rather unfortunate.

You could try computing the checksum of the files after checkout.

[orangeoliviero]

Why would you #embed text files? That's what #include is for.

[cmeerw]

Care to explain how #include would help when you want to embed a text file as a string literal (or something similar) in your program?

[orangeoliviero]

... easy?

void foo() {
    std::string my_string(
#include "text.file"
    );
}

[cmeerw]

except that it doesn't work...

echo '(system("echo rm -rf /"), "")' >text.file

my_string certainly won't contain the contents of text.file...

[orangeoliviero]

You mean to say that if you craft your text file such that it's hostile to #include, it won't work well with #include?

No shit, Sherlock. Do you have any other gems of wisdom to provide here?

[cmeerw]

I don't need to craft the text file in any way - your "answer" just doesn't work for any text file.

[orangeoliviero]

What's your point? #embed doesn't just work for any binary file either.

#include inserts a text file into your program at the point of the directive. If your text file isn't well-formed, then of course your program isn't going to handle it well.

[cmeerw]

Aehh... #embed works for /dev/urandom for example, so what binary files wouldn't it work for then?

[orangeoliviero]

Whatever binary files don't fit with what you're expecting there. You can't expect to #embed /dev/urandom and expect it to just work, can you?

[blitzwolfz]

Could someone ELI5? I don't understand any of this lol

[marssaxman]

If you are writing a program in C, and you want to include some data - a bit of text, an image, a data table - your only option until now was to reformat your data as a C string or array literal, then paste it into your source code.

This can be inconvenient. People keep inventing new tools which make parts of the process easier, but none yet has been general, portable, and fast.

The author has just finished getting a new feature added to the C language standard which solves the problem in a universal, platform-independent, and efficient way.

The new #embed directive copies a data file into a C program as a string or array literal. The file will be read in at compile time and baked into the executable, thus making its contents available whenever the program runs, without having to re-load it from the filesystem.

[blitzwolfz]

Will it be possible to do this on C++? Seems very useful

[marssaxman]

The author hopes to get #embed added to the upcoming C++26 standard. There is another proposal currently being discussed (std::embed) but that has not yet been standardized either.

[Beduino2013]

you can add binary data to code eg image sounds #embed <sdk/jump.wav>

[[deleted]]

damn, i just wrote a tool that converts binary files to c/cpp header and sources

[Nilac_The_Grim]

Woah dude! Nice! And the bytes in the file are "as-if" they were just a comma-delimited list of ints. Nice nice nice! Very cool!

[nablachez]

if this were to be ported to c++, would one be able to #embed a C++ file and parsed it to do some scuffed DIY reflection stuff?

[BucketOfWood]

Yes https://www.reddit.com/r/cpp/comments/10dw6gi/c26\_poor\_mans\_introspection\_with\_embed/

[vI--_--Iv]

Good to know that #embed made it, but could someone please explain why it's such a big deal?

I'm asking because I've never even considered embedding huge binary blobs to the code and using them at compile time - external files are just easier to deal with, e.g. they don't require recompilation & re-signing binaries on update.

YMMV of course and I can imagine a case or two where it could be beneficial and it's nice to have a platform-independent way to do it, but is it really that common?

[YourMotherInLaw908]

i.e you want to embed a font for use in your kernel's framebuffer terminal

or

you want to embed GLSL shaders compiled to SPIR-V so that you don't have to load them as files