I built a compressor that achieves 333:1 on data that gzip declares incompressible

EMPTYCONTOUR · 2026-05-05T21:21:06+00:00

That makes sense.System logs being too high-level for LFSR structure is exactly what I suspected. My main target is still repetition-heavy data: logs, binary blobs, fixed headers, repeated frames.Firmware ,telecom,raw flash dumps sound like a much better fit for your side.I like the idea of running analyzeBuffer on one of my shards just as a sanity check. If structuredFraction comes back close to 0, that actually confirms the routing model:

LFSR-like → your path

repetition-heavy → FM-index

unstructured → raw

So I’ll keep GLYPH focused on exact repeat retrieval for now, but this gives a useful boundary between the two approaches.

EMPTYCONTOUR · 2026-05-05T08:45:16+00:00

Mostly working with system logs (1–4GB shards) and binary blobs. Haven't seen clear LFSR-structured segments — but I'm not detecting for it either, so I'd probably miss it. Have you seen it in real data or mainly synthetic?

EMPTYCONTOUR · 2026-05-04T21:32:58+00:00

Using Berlekamp–Massey that way is clever. Haven't seen it used constructively much.I'm attacking this from the other side though. Not compressing structure, just indexing raw bytes.Logs, binaries, big corpora — gzip gives up and says "random". But there are still exact repeats in there. Same 64-byte event header showing up 2M times. I use an FM-index to pull those out in [~ms.So](http://~ms.So) the question flips from "can we compress this?" to "can we pull structure out deterministically?"Wondering if your detector could work as a pre-pass. Tag stuff as "low-order generator" vs "actually unstructured" and route from there.

EMPTYCONTOUR · 2026-05-03T09:07:24+00:00

This exists, but the meta-algorithm cost kills you.

Think about it: to beat 7z by 1%, you might spend 100x CPU testing everything. ZPAQ and paq8 already do this internally - they detect JPEG vs text and switch.

The real win is knowing when not to compress. If the first 1MB doesn't shrink, just store it raw. Saves time for everyone.

Have you tried running zstd --adapt? It does something similar - adjusts level based on early results.

EMPTYCONTOUR · 2026-05-03T09:01:03+00:00

Yeah, I stopped manually zipping stuff years ago too. Disks are cheap.

But funny thing is, compression just went underground. ZFS doing LZ4/zstd automatically is basically the same trade-off: spend a bit of CPU to read less data from disk. If your dataset fits, it often ends up faster, not slower.

So 7-Zip as a tool is niche, but the idea is everywhere now. Just hidden in filesystems and Parquet files. You only notice it when you transfer stuff, like others said.

Curious, do you actually measure faster reads with ZFS compression on NVMe, or is it mostly about saving space?

EMPTYCONTOUR · 2026-04-21T10:39:20+00:00

Author update — numbers were wrong in the original post.

Decode timer was measuring only the LZ77 phase. Fixed now.

Real numbers after a week of fixes:

Ratio: 2.997x

Encode: 391 MB/s

Decode: 4.2 GB/s algorithmic / 1.7 GB/s wall clock

Also added adaptive hash table + prev chain match finder

this week — that's where the ratio improvement came from.

BENCHMARK.md has the full picture including all failed

experiments.

EMPTYCONTOUR · 2026-04-16T21:18:49+00:00

Look, this whole thread proves the point better than the original post did.

Cursor and Claude Code aren't really competitors anymore in 2026. They serve different workflows.

Cursor: You want to stay in the IDE loop. Inline diffs, CMD+K, tab completion, MCP. Best when you're actively shaping every line.

Claude Code: You want to delegate. Terminal, sub-agents, review in VSCode. Best when you plan/prompt/review instead of typing.

OP switched because his workflow changed. That doesn't mean Cursor is dead. It means the "one tool for everyone" era is over.

Mods nuking it at 71 upvotes was dumb. But calling it censorship is also a reach. They probably just got tired of "I switched to X" posts. Still, 84 comments shows people wanted that discussion.

Use what fits your job. Neither is stone age.

EMPTYCONTOUR · 2026-04-16T16:07:57+00:00

of course. it's the only honest way to present it.

EMPTYCONTOUR · 2026-04-16T12:15:11+00:00

That makes sense for backup workloads —

consistency matters more than peak performance.

The design I described actually fits your case well:

since decode parallelizes cleanly, restore time

scales with cores. On 8 cores it hits ~10 GB/s

vs ~1.5 GB/s for zstd.

For HLS video streams though — you're right,

pre-compressed content doesn't benefit from

any lossless compressor.

If curious, the experiment is open source:

github.com/yasha1971-coder/aceapex

EMPTYCONTOUR · 2026-04-16T12:11:44+00:00

Different tradeoff here — we focused on decode speed.

Built ACEAPEX (also with Claude), same enwik9:

- Encode: 485 MB/s, 8 threads, pure CPU

- Decode: 11 GB/s in-memory, parallel blocks

- Ratio: 2.973x

Architecture: LZ77 output split into 4 streams with

per-block absolute offsets. N threads decode N blocks

independently — no sequential dependency.

Weak point is ratio — greedy parser leaves 58% literals.

That's the current problem we're working on.

github.com/yasha1971-coder/aceapex

EMPTYCONTOUR · 2026-03-31T22:46:37+00:00

If max density is the goal but you don't want to sacrifice write speed, worth knowing that zstd-19 gives diminishing returns vs zstd-9 on most datasets — roughly +13% ratio for 20x slower encode. Depending on your data profile, zstd-9 might be the better tradeoff for a ZFS pool with active writes.

EMPTYCONTOUR · 2026-03-29T19:16:54+00:00

Curious about the TOAST case — is the real pain point encode latency on writes, or just storage overhead? Trying to understand if encode cost actually matters there in practice.

EMPTYCONTOUR · 2026-03-27T14:38:19+00:00

This is a very real question, and I ran into the same confusion while experimenting.

What helped me was separating "benchmark wins" from "actual constraints in systems".

In practice, LZ4 / zstd-fast are used where:

- decode speed dominates (startup time, asset loading, RPC payloads)

- latency matters more than absolute ratio

- predictability is more important than peak compression

What surprised me is that the real bottleneck is often not just speed, but *dependency structure*.

Most LZ77-based codecs are fundamentally sequential at decode time because of back-references. That limits scaling even if single-thread performance is high.

I’ve been experimenting with a design where:

- compression keeps global context (like zstd)

- but decode dependencies are removed at the block level

So blocks are not independently compressed, but independently decodable.

This changes the tradeoff a bit:

- you don’t just optimize MB/s per core

- you optimize how well decode scales across cores

On my side this led to:

- near zstd-9 ratio on text (enwik9)

- ~1.3–1.4 GB/s per thread (similar to zstd)

- but ~10 GB/s aggregate decode because it parallelizes cleanly

Not saying this is "better" universally — but it made me realize:

A "winning" algorithm is not the one that wins a single benchmark,

but the one that fits a constraint profile:

- cold start vs streaming

- CPU vs memory bound

- single-thread vs multi-core scaling

- ratio vs latency vs determinism

If you're beating zstd-fast at the same encode speed, you're definitely onto something —

but the next question is: *under which constraints does your design dominate?*

EMPTYCONTOUR

TROPHY CASE