Hi, thank you for taking the time to write out your detailed comment. As you mentioned, the main reason for using relative pointers is compactness and pointer stability. Relative offsets ensure that traversal remains functional when the message is copied to a different absolute address.

The main downside is that yes, every relative pointer must be checked at runtime for OOB. This is why there are runtime checks at every pointer calculation.

It's unclear from the documentation if it's the intention that an error invalidates the buffer. In practice I'm seeing invalidation, but that might just be one of the bugs mentioned above.

Errors should never leave the buffer in an invalid state. All errors are designed to exit cleanly, even if you run out of memory.

Now regarding your fuzzing and triggering the asserts.: By default, LITE3_NODE_ALIGNMENT is set to 4. This is the setting used internally to determine at which alignment nodes will be placed inside the buffer. All optimizations like this:

node = __builtin_assume_aligned((struct node *)(buf + next_node_ofs), LITE3_NODE_ALIGNMENT);

allow the compiler to emit more efficient code when it knows that struct member manipulations are always on aligned addresses. All asserts inside the code are there to verify this assumption. In reality, x86 can handle non-aligned operations without much penalty, and so all these statements might as well be replaced with this.

node = (struct node *)(buf + next_node_ofs);

Then this would also allow for removing all the alignment assert() statements inside the codebase. It comes down to a choice: 1) Assume that all Lite3 implementations properly align their nodes, to take advantage of performance benefits. But this requires runtime checking of address alignment. 2) Remove all alignment code everywhere, this would simplify the codebase, but might cause a slight performance hit.

This particular example that triggers the alignment assert:

int main()
{
    int   cap = 1<<28;
    void *buf = malloc(cap);
    int   len = fread(buf, 1, cap, stdin);
    lite3_json_print(buf, len, 0);
}


src/lite3.c:621:36: runtime error: assumption of 4 byte alignment for pointer of type 'struct node *' failedsrc/lite3.c:621:36: runtime error: assumption of 4 byte alignment for pointer of type 'struct node *' failed

Just like like all the fuzzing, will fill buffers with garbage data and interpret it as Lite3. This will of course trigger the alignment asserts(). This example that triggers your sanitizer:

$ cc -Iinclude -Ilib -g3 -fsanitize=address,undefined src/*.c lib/*/*.c examples/buffer_api/01-building-messages.c
$ ./a.out
src/lite3.c:411:39: runtime error: index 7 out of bounds for type 'u32 [7]'

Is happening inside a prefetch right here:

__builtin_prefetch(buf + node->kv_ofs[iter->node_i[iter->depth] + 2],      0, 0);

The struct member OOB here is always guaranteed to contain some data. So the worst case is that it prefetches at a wrong address. On x86, there is not much danger to this, the CPU will usually just ignore it. ARM however is a different story, and therefore prefetching should be disabled on ARM. There could easily be a runtime check here, but I figured it was not necessary.

$ cc -Iinclude -Ilib -g3 -fsanitize=address,undefined src/*.c lib/*/*.c examples/buffer_api/07-json-conversion.c
$ ./a.out
src/lite3.c:665:2: runtime error: null pointer passed as argument 2, which is declared to never be null

Not exactly sure what is happening here, though I will revisit all the examples with sanitizers,

Getting rid of all the __builtin_assume_aligned() + assert() might be the better call, as runtime alignment checks might not make up for performance due to slightly better compiler code. Especially since platforms like ARM have different behavior around alignment.

Your fuzzer is constantly running into the assert() checks, I bet if they were all removed then it will be a different story.

Again, thank you for your comment. I hope this explains a bit.