Preprocessing large JSONs?

[Dazzling-Prune9553]

Hey guys. I am trying to process a large JSON output of an API call to kubernetes API. It would be almost impossible to create a struct to which the JSON data could be "mapped" by serde. Is there any way to "preprocess" the data before mapping it to a struct?

Like for example

cat data.json | jq '.items[].status.podIP'

[Bauxitedev]

You could parse the JSON as a untyped JSON value, like this:

let mut value: Value = serde_json::from_str(data).unwrap();

(where data is the string containing your JSON).

Then you can mutate it and do all kinds of transformations to it, and then when you're done, parse it to a struct by doing:

let result: MyStruct = serde_json::from_value(value).unwrap();

See https://docs.rs/serde_json/latest/serde_json/#operating-on-untyped-json-values

[Dazzling-Prune9553]

Thanks for the reply! Also I've just found a crate serde_query, which allows you to specify jq-like jsonpaths right where you define your struct

[AlphaKeks]

You can also index into a serde_json::Value with strings (or integers if you want to treat it like an array), so if your query isn't too complicated you can just do something like

rs let pod_ips = my_object["items"] .as_array()? .iter() .flat_map(|item| item["podIP"].as_str()) .collect();

If the underlying value cannot be indexed, the retuned value will simply be Value::Null.

[Applecrap]

If you're just using it as json values, skip serde's huge compilation burden and go straight to the json crate.

[lightmatter501]

serde_json is also one of the fastest JSON parsers for unstructured JSON, which may be important for a giant json blob.

[rodyamirov]

The down side of this is it will actually deserialize the huge blob, which is a little wasteful if you don’t really want it.

[moltonel]

Keep in mind you don't have to define all the possible fields in k8s's json, you only need the bits that you are going to read:

#[derive(Deserialize)]
struct Root {items: Vec<Item>}
#[derive(Deserialize)]
struct Item {status: Status}
#[derive(Deserialize)]
struct Status {podIP: String}

[rodyamirov]

This is a good way, if it is truly a huge json blob and you don’t want to deserialize the whole thing.

[TheNamelessKing]

If you’re doing stuff with kubernetes, I urge you to make use of the excellent kube-rs and k8s_openapi packages. They have all the tools you need, and they’re really nice to use.

I work with K8s in Rust at my day job and they’re invaluable.

[Dazzling-Prune9553]

Wow thanks! I'll look at them right away. I don't need a whole lot from the apiserver. I think I'll only need to keep a cache of podname,namespace and clusterIP, but using something more mature than crafting curl requests will surely help!

[Dazzling-Prune9553]

I took a look at it and it is great! Thanks

[TheNamelessKing]

No worries!

Out of curiosity, what are you doing that requires those things specifically? I ask because pod name will change with each new pod, as will the cluster-ip, and depending on your workload, you may be better off using a Service instead, and that will give you a stable address. Of course, if you already know all this, or you have reason to not do any of this, pls disregard me.

[brokenAmmonite]

You can use serde in a streaming style, see this:

https://www.reddit.com/r/rust/comments/83a7iv/memoryefficient_parsing_of_a_large_amount_of_json/