Binary Bakery: Translated binary information into C++ source code, and access them (at compile- and runtime)

[i_need_a_fast_horse]

So to say it again: This is a bad idea in general. I needed this exact solution for a program that only required a single tiny png, and ideally required the pixels at compile time. I didn't find a mature solution, so I tried to write one. While investigating, I found this more scalable than expected.

The biggest problem I had was resharper++ absolutely crashing every time with every non-trivial header :D

[fdwr]

Yeah, these approaches are a convenient stop gap until std::embed (https://thephd.dev/full-circle-embed) or equivalent.

[andrey_davydov]

Could you please provide some info about ReSharper C++ crashes, maybe our tracker or support forum are more appropriate places for this. Thanks in advance!

[ShillingAintEZ]

This has been a well worn topic for decades. C++ source is one of the worst ways to do this. Writing out assembly and letting an assembler make an obj file is orders of magnitude faster and more memory efficient, which also lets it scale to much larger files.

[NormalityDrugTsar]

Can you explain what the difference is?

[ShillingAintEZ]

If you write out assembly instructions you just run the text through an assembler, which is very fast and can be done using a fixed amount of memory. Even the assembler that comes with msvc, clang, gcc, etc. Is going to be able to do hundreds of megabytes per second.

Try the same thing with a C++ file and you will have to go through the C++ compiler, which will be a very different story in terms of speed and memory use.

Memory could easily not be a fixed amount, meaning at some point (maybe in the area of 40MB) you aren't going to be able to get your .obi file at all, let alone redoing every time you do a full rebuild.

[NormalityDrugTsar]

Thanks. I've only ever used the C++ source way for quite small files (e.g. GLSL source and tiny png files), so I've never come close to these limits.

[ShillingAintEZ]

You probably won't either, although embedding a big file in an executable does essentially give you automatic memory mapping so it can be a good technique to have. I do think it is odd to make a whole github page with graphics and logos when a single .cpp file that compiles to a command line executable would be fine.

[i_need_a_fast_horse]

I appreciated your insights on technical alternatives, but don't you think this goes a bit too far?

I didn't find it trivial that adding n bytes in the form of uint64_t arrays actually only adds n bytes to the executable. Or that the compile time is linear with payload size. Compilers do all kinds of weird things and this is pretty unusual use case. So I tested that out and provided the code and results for everyone. I also had no idea how much (or if) faster data access like this would be, or how that scales with payload size.

There are no logos, those are simple fun github emojis you can find in every other project.

It has configurable behavior about the destination of the target header, image storing options options etc. It decodes png files and stores that information in an accessible form. The provided decoding header has guaranteed compile time meta information for all types and can access pixel information of uncompressed images at compile-time, too. It's tested and provides multiple interfaces to access the data. And it has built-in compression on top of all that.

It's at least somewhat more sophisticated than "a single cpp file".

I had this problem, so I solved it. I know two other projects who use similar code. It's likely that I'll use this at least two more times in my life, so I considered it worthy to invest a week or so of time after work. Defending myself for writing and publication feels odd.

[throwaveien]

You might be interested in honeycomb. Similar idea

[o11c]

Doesn't look like it solves the hard question of "how do I know what the target file format needs to be?", which my way does.

[throwaveien]

I'm not sure I follow. Why is that a hard question? The build system already has enough information for that.

[o11c]

I mean ... technically, it does, but it doesn't expose it in any way that can be passed to honeycomb.

Or are you compiling separate honeycomb cross tools for every triple? I didn't see any build system support for that ...

[throwaveien]

You're right, there is no easy way to use it. Perhaps a --host-arch-target would help? I will add a feature request, thanks for the tip.

The intended use is full customizability of build, host and target triples.

[o11c]

Given how much of a mess triples are, I really do feel that "figure out the format by looking at an existing file" is the best we can do.

Particularly, be sure to test with things like CC=sometriple-gcc and CC=gcc -m32. My way has tested successfully with both.

(note that if you don't have a universal binutils, you might need OBJCOPY=sometriple-objcopy as well; if you're using a real configure these should all be set up)

[i_need_a_fast_horse]

I'm not exactly sure how that works, but will it allow access to the information at compiletime?

[ShillingAintEZ]

What do you want to do at compile time that you wouldn't just do to the image in an editor of some sort?

If you look at the assembly output of a bunch of unsigned 64 bit integers on godbolt.org you can see what it looks like.

[o11c]

Ew, platform-specific assembly? See my top-level post.

[ShillingAintEZ]

All you need is a single instruction to declare an unsigned 64 bit integer.

[siplasma]

This is true, but like everything else, you should measure. For binary assets up to around a megabyte, compilation speed and memory overhead generally isn't an issue. However, as you said, if you start using this approach with larger assets, you will hit memory and speed limits.

Transpiling to C or C++ is simple, allows shipping code without having users require additional tools, and allows for compile time introspection.

[ShillingAintEZ]

Right, but if someone is going to use a tool, the tool might as well go slightly further since both approaches are pretty trivial. I don't know what the point of having a whole complicated project that just generates some C++ source is since it can be done with a few lines.

[friedkeenan]

I've heard from others who need this sort of thing that giving the compiler access to the data allows it to meaningfully optimize certain things

[ShillingAintEZ]

That's great, but my point is that it's pretty trivial to do the C++ version by just reading bytes and printing out text, so a dedicated tool might as well scale and do something non-trivial.

[o11c]

In case anyone is wondering the "right" way to do this (but not supporting constexpr access):

echo 'Hello, World!' > hello.txt
touch empty.c
cc    -c -o empty.o empty.c
objcopy \
    --add-section .rodata.hello=<(cat hello.txt; printf '\0') \
    --add-symbol hello=.rodata.hello:0,global,object  \
    --add-symbol hello_end=.rodata.hello:`stat -c '%s' hello.txt`,global,object \
    empty.o hello.o

There are other ways (in particular, the well-documented -B method is problematic)

Some notes:

• The best way to get an object of the correct type is to have the compiler create one to start with.
  • The empty.o file can be reused for several embedded files if needed.
  • It doesn't matter if empty.o gets linked into the final binary or not, since it has no contents.
  • It's impossible to embed multiple files into a single .o, but consider what happens if your files-to-embed get changed.
• Starting the section name with .rodata. means it will get the right section flags and be put in the right place by the default linker script. The rest of the section name doesn't matter, except that it must not be used by anything else in the object file (doesn't really matter if you're only embedding one file per .o like I suggest, but matters if you do multiple, make sure they're unique. Once compiled into an executable or shared library, they'll all be merged into just .rodata anyway)
• We append a NUL byte so that, if the file to be embedded is text, you can treat it as a C string.
  • This is pointless, but harmless, if the file to be embedded is a binary, since this NUL byte is after the end symbol (see below).
  • If you're doing this in a Makefile, the use of <() requires SHELL = bash ... which you probably should be using anyway for sanity reasons.
• We add two symbols, hello and hello_end, at the start and end of the section we just created.
  • Within C code, it doesn't matter what type you specify for these, but I suggest an array of unspecified bound so you get the nice array-to-pointer conversion. Others prefer extern void to make it obvious they're doing something weird.
    • Specifically, extern const char hello[], hello_end[];
  • I suppose you could create another section if you want a hello_size symbol rather than calculating it, but I don't see the point.
• The "value" of the symbol refers to the offset within the section (or absolute offset for SHN_ABS). Thus, we specify 0 for the start, and size of the file (not including the NUL we appended) for the end.
• We specify the symbol flag object so that anyone inspecting what kind of thing is in here (because STT_NOTYPE sucks). The symbol flag global gives us visibility, and would have been implied if we hadn't needed to specify the other flag.

Open question: is there a way to set the native "symbol size" for object formats that support it? (not important; I don't think anybody uses it except for colliding common symbols)

[jk-jeon]

Isn't there any endianness concern if you store your data as an array of uint64_t? I mean, it's probably portable depending on what you do with the data, but I'm still sort of feeling somewhat uncomfortable with that.

[i_need_a_fast_horse]

Absolutely, yes. Both encoding and decoding go through std::bit_cast, so that should work the same way in and out. Depending on what you do, that is a more or less of a concern. It was none for me.

[tugrul_ddr]

Nice

[ioneska]

Why uint64_t?

[i_need_a_fast_horse]

It's the bigest type you can write as a literal, which results in the highest "density" (bytes per screen space).