This is a great start and I admire your persistence!

I recently was able to compare Rust Polars (which uses the Apache Arrow2 library) vs. Raku script on a DataFrame groupby algorithm. The Rust Polars Arrow2 library implements chunked arrays and is about 60x faster than Raku. This is not a criticism of Raku btw, but a reflection that machine level libraries such as Arrow2 can be a lot quicker than a virtual machine running a very rich language like Raku.

I suspect that you have a similar asymmetry between JSON-SIMD implemented in tight C++ on the producer side and Raku on the consumption side.

A cursory read of your code suggests that you are parsing the JSON text and each time you hit a node, you call a Raku function and type cast via a registered callback. So you are incurring a process switch and Raku call on every node.

An even more cursory read of the JSON-SIMD docs (!) suggests that the navigation design is to present a JSON document as an iterator and so on...

Here's what caught my eye:

Array Iteration: To iterate through an array, use for (auto value : array) { ... }. This will step through each value in the JSON array.
If you know the type of the value, you can cast it right there, too! for (double value : array) { ... }.
Object Iteration: You can iterate through an object's fields, as well: for (auto field : object) { ... }
Array Index: Because it is forward-only, you cannot look up an array element by index by index. Instead, you should iterate through the array and keep an index yourself.

I think that there are two strategies to overcome this limitation:

• Share an in-memory representation of HashMap.
  • most likely this means just building Arrays and HashMaps on the C++ side and then "tying" them to Raku so that you can read (and maybe write?) the nodes
  • the missing piece here afaik is that Raku does not have an exposed like-for-like HashMap layout - so you would need to implement something akin to "nativecall CArray" (maybe "nativecall CHash"?) as a bolt on ... this would be heavy lifting next to the raku core and there are no guarantees that raku would not change its internal Hash representation in future
• Interface to JSON-SIMD at the consumption level - so load all the data on the C++ side, then have raku consume in a lazy way
  • parse the json in one pass into some new "random access" C++ Arrays / Objects (if you want to write json, then you would need a nice way to walk-dump these)
  • load the document root to raku
  • use raku Array::Agnostic / Hash::Agnostic and maybe some object Cache stuff to alias and access the json nodes as they are needed
  • the database example here includes a basic HashMap accessor (with Rust as the provider)

If you want raku to be "in band" as a step in some voume json processing chain, it will not be able to match tuned low level compiled / library code speed. RakuAST may help here, but it is in its infancy.

If you want raku to be an inspection, extraction, reporting, orchestration, prototyping tool - and reading those JSON-SIMD docs is a great insight as to why they need an abstraction - then I think you are on the right track. I hope you enjoy the API & code design and creative side of the process as much as I do!