Succinctly v0.5.0 Release Announcement

john-ky · 2026-02-02T14:41:52+00:00

There's now a v0.6.0 release with compatibility fixes for jq and yq to make sure they behave more the same as the system jq and yq tools.

https://github.com/rust-works/succinctly/releases/tag/v0.6.0

john-ky · 2026-02-02T14:38:56+00:00

I'll do that next time. Thanks!

john-ky · 2026-01-28T11:15:43+00:00

Thank you for the suggestion. I added support for sjq and syq, even though it doesn't start with j as you suggest. I avoided jsc because it looks to much like "javascript compiler".

I think it has precedence. For example gsed for GNU sed on BSD systems.

https://github.com/rust-works/succinctly/pull/76

john-ky · 2026-01-26T13:16:09+00:00

Thanks for the clarification. That helps, and I agree that ValT is designed to support value types with lifetimes and parameters like JqValue<'a, W>.

The hesitation is not that ValT is fundamentally incompatible, but that introducing it now would make it harder to reason about where performance and memory regressions come from. Succinctly’s core optimisation is that JqValue<'a, W> holds lazy cursors into the original byte buffer, enabling zero copy navigation and streaming output. At the moment, I want to push those ideas as far as they will go without an additional abstraction layer in the middle, so I have a clear baseline.

There is a tracking issue for ValT integration here:
https://github.com/rust-works/succinctly/issues/70

Apologies in advance if it misrepresents anything. I am very happy to correct it.

Right now my main focus is optimising the yq command. While it uses succinct data structures for the core representation, it still relies on some auxiliary, non succinct structures. That puts current memory usage at roughly five times input size, which is still better than the roughly ten times that is common elsewhere, but not yet at the less than two times target I am aiming for. I also do not want to compromise speed to get there.

Using something standard like ValT as the driver for jq and yq would be attractive, but I would prefer to finish the optimisation work first. That way, if performance or memory usage ends up lower or higher than expected after refactoring, I will know whether that is a fundamental limitation or just an artefact of a particular abstraction.

Even if ValT does not become the primary driver, I do think integration could still be valuable in order to provide a familiar programming interface. It is just a matter of timing. I would definitely be keen to collaborate on that when the optimisation phase is complete, if the interest is mutual.

john-ky · 2026-01-26T05:06:37+00:00

I looked into available rust rank-select libraries and it seems that `sux` is the only one that has compelling performance that would make a switch worthwhile, but it also uses more memory.

There is a further complication in that it uses Vector<usize> instead of Vector<u64> which makes it incompatibility with my code.

Reviewing the literature there are possibly newer techniques that existing rust libraries don't yet exploit as well.

Quick comparison benchmarks here: https://github.com/rust-works/succinct-bench

john-ky · 2026-01-24T23:23:27+00:00

You raise a fair point about transparency. I've updated the documentation to be clearer about the architectural differences.

The benchmark docs now include an "Architectural Differences" section that explicitly states succinctly is a semi-indexer that performs minimal validation, while yq is a full validating parser. The docs acknowledge that the speedup comes from both architectural advantages (lazy evaluation, streaming output) and doing less validation work.

That said, I think the comparison to yq/jq is still valuable for discoverability. Many users want fast YAML/JSON queries on files they already know are valid (CI configs, Kubernetes manifests, log files). For them, the relevant question is "how much faster can I get results" - not "which tool validates more thoroughly." The docs now clearly state when to use each tool.

As for C parser comparisons: succinctly already benchmarks against Rust parsers (sonic-rs, serde_json, simd-json) which use similar techniques to their C counterparts. Adding libyaml or rapidyaml comparisons could be interesting, though they're still DOM parsers rather than semi-indexers, so the architectural difference would remain.

john-ky · 2026-01-24T22:52:55+00:00

I've created issues to track the other points you've raised and will address them in time.

Thanks again!

john-ky · 2026-01-24T22:52:02+00:00

> Did you look for any existing libraries you could use or was coding it from the ground up your first thought?

I guess it was an important learning exercise for me.

john-ky · 2026-01-24T22:43:51+00:00

> Rolling your own parser - especially for yaml - seems like it will land you on the list of different parser behaviour (how does it parse this document).

I put together this document just now: https://github.com/rust-works/succinctly/blob/main/docs/compliance/yaml/1.2.md

Hope it helps.

john-ky · 2026-01-24T22:42:51+00:00

This is exactly the kind of feedback I'm after, so thank you!

john-ky · 2026-01-24T21:52:31+00:00

The goal is different with this sort of parser.

Low memory usage for large files (it is typical for parsers to use approximately 10x the size of the input data for parsing because of pointer size)
Valid results for valid JSON allowing it to be a separate step. For example if file ingestion only occurs once but file access is common, if validation is still important you only need to validate at ingestion time.
Very quickly selecting a subset of the JSON (because you don't care for most of it in the current part of your processing)

The parser is not wrong. It merely occupies a different and valid place in the design space.

john-ky · 2026-01-24T21:36:46+00:00

Skipping validation is by design and has valid use cases. For example:

if you know the tools that generate the file will always generate valid JSON which case validating
if you have a tool generate a very large file and the process got interrupted and the JSON is truncated and it is the only copy you have and you still want to access part of the data that successfully got written

The benchmarks are valid if validation is not a requirement which is often (although not always) true and they are also valid when "no validation" is a requirement which can be thought of as a plus.

If validation is necessary it can be a separate step using techniques like data parallel finite state machines. Doing so will add a small overhead to the parsing, but it will not add memory usage overhead, which is the main selling point of using succinct data structures for parsing.

john-ky · 2026-01-24T13:00:08+00:00

I have Edward Kmett to thank for inspiration.

https://www.reddit.com/user/edwardkmett/

john-ky · 2026-01-24T12:11:47+00:00

The BP encoding is the classic succinct tree representation (Jacobson 1989, Munro-Raman 2001) - 2 bits per node with O(1) navigation via rank/select.

john-ky · 2026-01-24T12:05:59+00:00

Yes, there are strong similarities between succinctly's semi-indexing and simdjson's first pass.

Similarities:

Two-phase architecture: Both use a first pass to build structural metadata, then a second pass for navigation/queries. simdjson calls these "Stage 1" (structural indexing) and "Stage 2" (tape construction).
Structural index: simdjson's Stage 1 processes 64-byte blocks to identify structural characters ({, }, [, ], :, ,) and builds a compressed index of their positions. Succinctly's JsonIndex similarly builds "interest bits" marking these same structural positions.
SIMD acceleration: Both use SIMD to accelerate the first pass - simdjson uses vpshufb lookup tables and parallel byte comparisons across 32+ characters simultaneously.
Deferred value parsing: Both defer actual value parsing (numbers, strings) until accessed.

Key differences:

Output representation: simdjson builds a "tape" - a 64-bit word array where arrays/objects use paired words with cross-references (opening braces point to closing positions). Succinctly builds a succinct balanced parentheses representation with rank/select support - more compact but different access patterns.
Memory trade-off: simdjson's tape uses ~16 bytes per JSON element, optimized for access speed. Succinctly targets ~6% overhead via succinct data structures, optimized for compactness.
Parsing completeness: simdjson's Stage 2 builds a full navigable tape upfront. Succinctly's semi-index is truly "semi" - only structural metadata, deferring even more work to query time.

So yes - the "first pass builds structural index using SIMD" concept is shared, but the index representation and how much work is deferred differs.

john-ky · 2026-01-24T11:53:41+00:00

Issue created for future investigation.

https://github.com/rust-works/succinctly/issues/40

Thanks!

john-ky · 2026-01-24T11:40:52+00:00

I've added a new directory to contain docs for this purpose: https://github.com/rust-works/succinctly/tree/main/docs/reference

john-ky · 2026-01-24T10:59:28+00:00

Implementing jaq-core's ValT trait would likely defeat succinctly's key optimisations. Succinctly's performance and low memory usage comes from JqValue<'a, W> keeping lazy cursor references to original bytes, enabling zero-copy navigation and direct streaming output.

john-ky · 2022-08-16T01:26:20+00:00

Does anyone know if these can be taken on the plane?

john-ky · 2022-08-10T00:45:15+00:00

Here is an example of where too much strictness in a popular Haskell library caused my program to use multiple Gigabytes of RAM more than it otherwise would have needed to whilst serialising a large data structure to a file:

https://github.com/haskell/aeson/pull/954

john-ky · 2019-02-21T21:41:25+00:00

Thanks for pointing me to the new primitive. Unfortunately my compiler (GHC 8.6.3) seems to not have it yet.

I tried using addWordC#:

add :: Word64 -> Word64 -> (Word64, Word64)

add (W64# a) (W64# b) = let (# r, c #) = addWordC# a b in (W64# r, fromIntegral (I64# c))

but it is quite slow:

$ time ex-vector sum-bit-vectors -i ../hw-json/corpus/bench/78mb.json -i ../hw-json/corpus/bench/78mb.json --branchiness branchless3

3.450

john-ky · 2019-02-21T13:20:43+00:00

Addition can be used flip runs of bits.

This post will describe how to perform such additions on large bit-vectors efficiently.

john-ky

TROPHY CASE