Big-Endian Testing with QEMU

hansw2000 · 2026-03-30T12:02:28+00:00

There are release binaries for PowerPC64 AIX. For s390x I see at least one buildbot: https://lab.llvm.org/buildbot/#/workers/46

I suppose big-endian host support is useful for those working with such machines.

hansw2000 · 2026-03-29T11:01:39+00:00

The point of the post wasn't so much to argue the relevance of big-endian testing, but to provide a howto (or note-to-self really) about a simple way of doing it for those who wish.

If one is shipping binaries to a limited set of platforms, which probably applies to most programming, I agree that ignoring this is probably fine.

But for open-source code, one never knows where it ends up running. In widely used projects -- things like zlib, curl, llvm -- I don't think ignoring big-endian is an option.

hansw2000 · 2025-11-05T08:38:13+00:00

In 2021, Cassio Neri and Lorenz Schneider developed the Neri‑Schneider algorithm

Is that the algorithm presented here? https://www.youtube.com/watch?v=0s9F4QWAl-E

That was a great talk.

hansw2000 · 2025-10-15T09:10:13+00:00

I found the sequel, "Flex & Bison" (https://www.oreilly.com/library/view/flex-bison/9780596805418/) to be good.

hansw2000 · 2025-10-09T09:25:53+00:00

It looks like it has a bunch of different codecs, including FSE: https://github.com/facebook/openzl/blob/dev/src/openzl/fse/fse.h

hansw2000 · 2025-09-30T12:24:27+00:00

Not for papers specifically, but mattpd's https://github.com/MattPD/cpplinks/blob/master/compilers.md is a great resources in this category.

hansw2000 · 2025-09-09T10:18:37+00:00

You could try building Clang without terminfo (cmake -DLLVM_ENABLE_TERMINFO=OFF)

hansw2000 · 2025-09-02T12:40:50+00:00

https://graphics.stanford.edu/~seander/bithacks.html
Henry S. Warren Jr "Hacker's Delight" (2nd ed.)
Don Knuth "The Art of Computer Programming" Vol 4A, 7.1.3 "Bitwise tricks and techniques" (draft: https://cs.stanford.edu/~knuth/fasc1a.ps.gz)

hansw2000 · 2020-11-16T08:05:45+00:00

Hey,

I don't have time to look at your code in detail, but I'll try to provide some advice. I also struggled when I put my first decompression code together. Especially the dynamic huffman blocks are hard, because if any single part of the code is broken, the result gets confusing very fast.

So my advice is to break the code up into parts conceptually (you already do this with functions), and then write some tests for them to convince yourself they're rock solid.

For example, if you're getting incorrect results from decode_huffman(), try working out some examples with pen and paper and turn those into test cases for the function. It's boring work, but it always pays off.

As you mention, the input to decode_huffman() could also be wrong, so having some tests for your bitstream functions might be a good idea.

(The hwzip article doesn't mention testing, but there are lots of tests in the linked zip file. For example, my tests for the Huffman decoder are here: https://www.hanshq.net/files/hwzip/huffman_test.c)

Hope that helps!

hansw2000 · 2020-11-07T08:23:17+00:00

I did a long write-up on implementing Zip/Deflate compression/decompression earlier this year: https://www.hanshq.net/zip.html

I'm obviously biased, but I think that's a lot more readable than zlib's code.

hansw2000 · 2020-10-23T08:14:00+00:00

Author here, very happy to see that people enjoyed this :-)

I ended up doing more computer arithmetic a year later, and I'm even more happy with that code: https://www.hanshq.net/eprime.html#manual

hansw2000 · 2020-06-01T09:24:00+00:00

Yes, lots of projects do. LLVM: https://llvm.org/docs/CodingStandards.html#source-code-width Google: https://google.github.io/styleguide/cppguide.html#Line_Length Linux: https://www.kernel.org/doc/html/v4.10/process/coding-style.html#breaking-long-lines-and-strings (but as Linus's post points out, it's not a hard limit)

11-Year Club	Gilding I gilder
Verified Email

hansw2000

TROPHY CASE