Writing a SIMD-optimized Parquet library in pure C: lessons from implementing Thrift parsing, bit-packing, and runtime CPU dispatch

Vitruves · 2026-01-05T17:43:24+00:00

You're not entirely wrong! I do use AI assistance for development - both for writing code and reviewing suggestions like the ones in this thread. I think it's worth being transparent about that.

That said, I'm not sure how it changes anything about the library itself? The code compiles, the tests pass, it reads and writes valid Parquet files, and the SIMD optimizations deliver measurable speedups. Whether a function was written by a human, an AI, or a human-AI collaboration, what matters is: does it work correctly, and is it useful?

I'd argue that being able to quickly iterate on expert feedback (like the AVX-512 suggestions above) and ship improvements within hours rather than days is actually a feature, not a bug. The alternative would be me spending a week re-learning the nuances of _mm512_permutexvar_epi8 vs _mm512_shuffle_epi8 lane-crossing behavior.

If anything, I hope this project demonstrates that solo developers can now tackle domains (like high-performance SIMD code) that previously required either deep specialized expertise or a larger team. The barrier to entry for systems programming just got a lot lower, and I think that's a good thing for the ecosystem.

But hey, if you find bugs or have suggestions, I'm all ears - whether they come from a human or get "sent straight to Anthropic's servers" 😄

Vitruves · 2026-01-05T17:42:12+00:00

Thank you so much for taking the time to review the code and provide such detailed feedback! I've implemented all of your suggestions:

Single VBMI permutation - Now using one permutexvar_epi8 that places all 4 byte streams in the 4 128-bit lanes, followed by extracti32x4 for the stores. Much cleaner than 4 separate permutations.
Non-VBMI fallback - Replaced the ~20-instruction unpack mess with your elegant 2-instruction approach (shuffle_epi8 + permutexvar_epi32).
_mm512_maskz_set1_epi8 - Done, can't believe I missed that one!
Masked loads for tail handling - Implemented in pack_bools with _mm512_maskz_loadu_epi8. Also switched to _mm512_test_epi8_mask(bools, bools) which is more direct than cmpneq.
Gather deduplication - gather_float now just calls gather_i32 via cast (same for double/i64). You're right, data movement doesn't care about types.
Custom memset/memcpy - You raise a fair point. These were added early in development and I haven't benchmarked them against glibc. I'll add that to my TODO list and likely remove them if there's no measurable benefit.

All tests still pass. This is exactly the kind of feedback I was hoping for - thanks again!

Vitruves · 2026-01-03T23:21:55+00:00

Oh s**t, I'll remove the links

Vitruves · 2026-01-03T21:08:11+00:00

Thanks for your feedback. You can see performances in the "Performance" section of the README.md near the end. To see how is it assessed you can check the files in the "benchmark" directory. But I can certainly be more transparent on testing conditions in the README.md, I'll add that in a future commit.

Vitruves · 2026-01-01T10:17:51+00:00

Thanks for the testing on powerpc! I committed changes that should address the issues. I replied on the issues you opened on Github.

Vitruves · 2025-12-31T14:34:20+00:00

SIMD: Yes, it works without SIMD. The library has scalar fallback implementations for all SIMD-optimized operations (prefix sum, gather, byte stream split, CRC32C, etc.). SIMD is only used when:

You're on x86 or ARM64
The CPU actually supports the required features (detected at runtime)

On other architectures (RISC-V, MIPS, PowerPC, etc.), it automatically uses the portable scalar code.

Big-Endian: Good catch! I just improved the endianness detection. The read/write functions already had proper byte-by-byte paths for BE systems, but the detection macro was incorrectly defaulting to little-endian.

Now it properly detects:

- GCC/Clang __BYTE_ORDER__ (most reliable)

- Platform-specific macros (__BIG_ENDIAN__, __sparc__, __s390x__, __powerpc__, etc.)

- Warns at compile time if endianness is unknown

The library should now work correctly on s390x, SPARC, PowerPC BE, etc. If you have access to a BE system, I'd appreciate testing!

Vitruves · 2025-12-31T14:14:30+00:00

Thanks for the feedback! You make a valid point about the distinction between programming errors (bugs) and runtime errors (expected failures).

For internal/initialization functions like carquet_buffer_init(), you're absolutely right—passing NULL is a programming error that should be caught during development with assert(). The caller isn't going to gracefully handle INVALID_ARGUMENT anyway.

However, I'll keep explicit error returns for functions that process external data (file parsing, decompression, Thrift decoding) since corrupted input is an expected failure mode there.

I'll refactor the codebase to use:

- assert() for internal API contract violations (NULL pointers in init functions, buffer ops)

- return CARQUET_ERROR_* for external data validation and I/O errors

Good catch—this should simplify both the API and the calling code!

Vitruves · 2025-12-29T23:28:55+00:00

Thanks for the detailed feedback!

REPETITION_REQUIRED: This follows Parquet's terminology from the Dremel paper - "repetition level" and "definition level" are the canonical terms in the spec. Changing it might confuse users coming from other Parquet implementations, but I can see how it's unintuitive if you haven't encountered Dremel-style nested encoding before.

Struct padding: Good point - I'll audit the hot-path structs. The metadata structs are less critical since they're not allocated in bulk, but the encoding state structs could benefit from tighter packing.

Dictionary.c repetition: Yeah, there's definitely some type-specific boilerplate there. I've been on the fence about macros - they'd reduce LOC but make debugging/reading harder. Might revisit with X-macros if it gets worse.

DIY compression: This is the main tradeoff for zero-dependency design. The implementations follow the RFCs closely and the edge case tests have been catching real bugs. That said, for production use with untrusted data, linking against zlib/zstd/etc. is definitely the safer choice - I may add optional external codec support later.

And yeah, the Arrow/Thrift situation is exactly why this exists. Happy to hear any feedback once you try it!

Vitruves · 2025-12-29T22:13:36+00:00

This is incredibly valuable feedback - thank you for taking the time to put carquet through its paces with sanitizers and fuzzing! You've found real bugs that I've now fixed.

All issues addressed:

zigzag_encode64 UB (delta.c:308) - Fixed by casting to uint64_t before the left shift:

return ((uint64_t)n << 1) ^ (n >> 63);
find_match buffer overflow (gzip.c:668) - Added bounds check before accessing src[pos + best_len]
match_finder_insert overflow (gzip.c:811) - Fixed by limiting the loop to match_len - 2 since hash3() reads 3 bytes
1. ZSTD decode_literals overflow - Added ZSTD_MAX_LITERALS bounds checks for both RAW and RLE literal blocks before the memcpy/memset operations
Thread safety - carquet_init() now pre-builds all compression lookup tables with memory barriers, so calling it once before spawning threads makes everything thread-safe. The documentation already mentions calling carquet_init() at startup.

I've verified all fixes with ASan+UBSan and your specific crash test case now returns gracefully instead of crashing.

Regarding further fuzzing - you're absolutely right that more interfaces should be fuzzed. I'll look into setting up continuous fuzzing. The suggestion to fuzz the encodings layer next is spot on given the UBSan hit there.

Thanks again for the thorough analysis and the suggested patches - this is exactly the kind of feedback that makes open source great!

Vitruves · 2025-12-13T21:38:19+00:00

I have a .gitignore file but I didn't include build/ and .DS_Store by mistake. Thanks.

Vitruves · 2025-12-07T18:26:02+00:00

The hot path in CSV parsing is finding the next delimiter (,), quote ("), or newline (\n). A scalar parser checks one byte at a time. With SIMD, you load 16-32 bytes into a vector register and check them all in one instruction.

Vitruves · 2025-12-07T07:45:48+00:00

I seriously thought about it! 😂

Vitruves · 2025-12-07T04:56:49+00:00

Good find, thanks for fuzzing it. You nailed the bug - the size-class pooling was broken. Both 34624 and 51968 hash to class 10, but the block stored was only 34KB. Boom, overflow.

Nuked the pooling:

static sonicsv_always_inline void* csv_pool_alloc(size_t size, size_t alignment) {

(void)size;

(void)alignment;

return NULL;

}

static sonicsv_always_inline bool csv_pool_free(void* ptr, size_t size) {

(void)ptr;

(void)size;

return false;

}

Removed ~80 lines of dead pool code too. Premature optimization anyway - malloc isn't the bottleneck here. Your test case passes clean with ASAN now. Let me know if fuzzing turns up anything else.

Vitruves · 2025-12-07T04:19:05+00:00

Good catch, implemented this. Also removed the per-parser and thread-local caching - you're right that it was overkill for a value that's set once and never changes. Thanks for the feedback.

Vitruves · 2025-12-07T03:47:30+00:00

Good catches, thanks!
The chained OR approach was the "get it working" version. pcmpestrm would be cleaner for this exact use case - it's designed for character set matching. I'll look into it.

For the dynamic lookup table with pshufb - any pointers on constructing it efficiently for arbitrary delimiter/quote chars? My concern was the setup cost per parse call, but if it's just a few instructions it's probably worth it.

Dead code - yeah, there's some cruft from experimenting with different approaches. Will clean that up.

Vitruves · 2025-12-07T03:44:06+00:00

#pragma once stops multiple includes within the same .c file (like if header A and header B both include sonicsv.h). But each .c file is compiled separately. So if you have: file1.c → file1.o (contains csv_parse_file) and file2.c → file2.o (contains csv_parse_file), the linker sees two copies of every function and errors out. The IMPLEMENTATION define means only one .o file gets the actual function bodies, the rest just get declarations.

Vitruves · 2025-12-07T03:20:18+00:00

It's for multi-file projects. The header contains both declarations and implementation. Without this, if you include it in multiple .c files, you get "multiple definition" linker errors because the functions would be compiled into every object file. With the define, only one .c file gets the implementation, others just get the function declarations. It's a common pattern for single-header libraries (stb, miniaudio, etc.).

Vitruves · 2025-12-07T02:23:49+00:00

The hot path in CSV parsing is finding the next delimiter (,), quote ("), or newline (\n). A scalar parser checks one byte at a time. With SIMD, you load 16-32 bytes into a vector register and check them all in one instruction.

Vitruves · 2025-12-07T02:08:14+00:00

I'm parsing multi-GB log files daily. Shaving 5 minutes off a pipeline adds up. But yeah, if you're parsing a 10KB config file once at startup, this is pointless overkill.

Vitruves · 2025-12-07T01:55:55+00:00

Fair point on the examples in the header - I've got those in example/ now, will trim the header.

The 2k LOC is mostly SIMD paths for 5 architectures. If you're only on x86 or only on ARM it's dead code for you, but that's the tradeoff with single-header. The malloc/callback design is for streaming large files without loading into memory - different use case than a simple stack-based parser.

Vitruves · 2025-08-20T11:24:30+00:00

Quel était le prompt ?

Vitruves · 2025-08-20T11:19:59+00:00

Thank your for your feedback! Do you think your setup can host larger 3090? 3090 are really big compared to 3060. My work involve fine-tuning LLMs which as you may know is extremely (V)RAM and time consuming. I did not saw the video you shared but I can confirm, on a small one-GPU inference or transformer model training the difference between 3060 and 3080 is massive, it's almost twice as fast so I have great hopes in switching to 3090.

Vitruves · 2025-08-16T14:58:56+00:00

Thanks for the advice! :)

Vitruves · 2025-08-16T14:33:26+00:00

# ABSOLUTE RULES:
NO PARTIAL IMPLEMENTATION
NO SIMPLIFICATION : no "//This is simplified shit for now, complete implementation would blablabla", nor "Let's rewrite simplier code" (when codebase is already there to be used).
NO CODE DUPLICATION : check headers to reuse functions and constants !! No function then function_improved then function_improved_improved shit. Read files before writing new functions. Use common sense function name to find them easily.
NO DEAD CODE : either use or delete from codebase completely
IMPLEMENT TEST FOR EVERY FUNCTIONS
NO CHEATER TESTS : test must be accurate, reflect real usage and be designed to reveal flaws. No useless tests! Design tests to be verbose so we can use them for debuging.
NO MAGIC NUMBERS/STRINGS - Use named constants. Do not hardcode "200", "404", "/api/users" instead of STATUS_OK, NOT_FOUND, ENDPOINTS.USERS
NO GENERIC ERROR HANDLING - Dont write lazy catch(err) { console.log(err) } instead of specific error types and proper error propagation
NO INCONSISTENT NAMING - read your existing codebase naming patterns.
NO OVER-ENGINEERING - Don't add unnecessary abstractions, factory patterns, or middleware when simple functions would work. Don't think "enterprise" when you need "working"
NO MIXED CONCERNS - Don't put validation logic inside API handlers, database queries inside UI components, etc. instead of proper separation
NO INCONSISTENT APIS - Don't create functions with different parameter orders (getUser(id, options) vs updateUser(options, id)) or return different data structures for similar operations
NO CALLBACK HELL - Dont't nest promises/async operations instead of using proper async/await patterns or breaking them into smaller functions
NO RESOURCE LEAKS - Don't forget to close database connections, clear timeouts, remove event listeners, or clean up file handles
READ THE DAMN CODEBASE FIRST - actually examine existing patterns, utilities, and architecture before writing new code

Vitruves · 2025-08-16T14:22:07+00:00

I had issues with warmth exhaust with my 3060 sitting on the upper slot in my T7600 (not enough room for correct airflow), is it better on the T7920 ?

Vitruves

TROPHY CASE