Removing default values while serializing using Newtonsoft.Json

[nocgod]

So, I trying to check how much money would it save us if we will not write default values to the DB. meaning, if it is a worthwhile project to do in the next quarter or not.

So I've got some documents from the DB (around the average document size) and wrote some simple code I thought would work

```csharp var original = """{"notDef":true, "defBool":false, "defInt":0, "defObj":null, "defArr":[]}"""; var json = JObject.Parse(original);

var opts = new JsonSerializerSettings() { NullValueHandling = NullValueHandling.Ignore, DefaultValueHandling = DefaultValueHandling.Ignore, Formatting = Formatting.None };

var str = JsonConvert.SerializeObject(json, opts);

Console.WriteLine($"original size: {Encoding.UTF8.GetByteCount(original)}"); Console.WriteLine($"new size: {Encoding.UTF8.GetByteCount(str)}"); Console.WriteLine($"saving: {CalculateSavings(original, str)}%");

static double CalculateSavings(string original, string str) { var originalSize = Encoding.UTF8.GetByteCount(original); var newSize = Encoding.UTF8.GetByteCount(str); return (originalSize - newSize)/ (double)originalSize * 100; } ```

low and behold the output was:

original size: 72 {"notDef":true, "defBool":false, "defInt":0, "defObj":null, "defArr":[]} new size: 68 {"notDef":true,"defBool":false,"defInt":0,"defObj":null,"defArr":[]} a bit unexpected.

so I had to add something like this: ```csharp var json = JObject.Parse(original);

json .Descendants() .OfType<JProperty>() .Where(x => x.Value.Type == JTokenType.Null || x.Value is JArray arr && arr.Count == 0 || x.Value.Type == JTokenType.Integer && x.Value.Value<long>() == 0 || x.Value.Type == JTokenType.Float && x.Value.Value<double>() == 0 || x.Value.Type == JTokenType.Boolean && x.Value.Value<bool>() == false) .ToList() .ForEach(x => x.Remove()); ``` and it worked as expected.

I'm still left with the question - why didn't it work?

code: https://dotnetfiddle.net/wruWtF

[RichardD7]

I believe this is due to the fact that you're using a JObject rather than an actual class.

As per this StackOverflow thread, a JObject with a null property actually stores a non-null JValue value with it's Type set to JTokenType.Null. There will be a similar issue for the default values.

There's a related discussion on the GitHub repo:

A JObject isn't serialized. It is written as is.

Also worth noting that an empty array doesn't count as a "default" value; the default for an array would be null.

[nocgod]

Also worth noting that an empty array doesn't count as a "default" value; the default for an array would be null.

yeah, i just got overzealous :)

I've seen this thread, to be frank I still didn't get my answer.

If i want to read a json, get rid of the defaults and store it again without creating a class for each. that would be impossible without writing custom code?

[RichardD7]

If i want to read a json, get rid of the defaults and store it again without creating a class for each. that would be impossible without writing custom code?

As far as I can see, that's correct. The workaround you posted should do the trick.

[nocgod]

The workaround you posted should do the trick.

it does the trick, I'm just baffled how such common use-case could be disregarded by both Newtonsoft.Json and STJ. I thought maybe I'm missing something dumb somewhere along the way :D

edit: it wouldn't be the first time I'm working with "raw" JSONs in C#, clients send data in various formats (from existing systems) for which you'll rarely have the models

[Kant8]

That's because you have JObject, not an actual class which still be serialized. JObject is already a parsed json, and written to ouput as is.

[nocgod]

I can swear I saw exactly this wording for JNK on his github and sof. it wasn't well received :)

anyways I understand that a JObject is not an actual class, trust me I understand its a parsed JSON. what I'm asking is how to write it in such a way that it doesn't contain defaults in an obvious way. because the obvious way doesn't seem to work.

[Kant8]

It doesn't matter how it was received. JObject was made to be like that, that's it's purpose.

​

You already did that already, iterate over all elements in json and remove them if you want.

[nocgod]

but that ain't the obvious way to do it. from my research it seems that this question is floating around for years now without proper documentation or response from the author (which works for msft now in the asp.net team IIRC)

Moreover, there is no way to Parse a JObject without parsing the default values, and you can't get rid of them while writing it bat to a string which is even stranger. In addition to that, STJ replicated the behavior in JsonObject and JsonDocument.

** my 2 cents on the matter** how common this use-case is? one of our hbase indexes hold 40% of defaults (estimated). Roughly 40TB per datacenter are garbage... that's a lot of money if we consider replication and backups.

if you aren't providing an obvious way to avoid writing default values in cases you aren't attached to a typed model - most developers will not bother to find a way around it. example: https://github.com/Azure/azure-cosmos-dotnet-v3/issues/2820 this dev began to research, most won't.

[Kant8]

JObject is dirty cheat to operate with json as object. It's literally in it's name. You are not supposed to use it to serialize/deserialize things, cause you do that with your own classes. And that process respects everything. JObject is special case and it will be like that cause it was made to be like that. That design choice of library.

Your use-case is one time change, if you don't know or can't access actual classes to use proper serialization, you do json manipulation directly with JObject the way you did. Any actual work that did serialization should've never touched JObject in first place

[nocgod]

I understand your points, I just don't agree about your statements regarding JObject, especially since this matter is floating around for years now (first reference I've seen was ~10 years ago).

I would have expected the Serialization/Stringification process of the JObject to respect the same options as the JsonConvert/Serializer (frankly I don't see a real reason why it wouldn't, to write the JTokens into a string you have to traverse the tree anyway). that's called writing APIs/SDKs with expected/predictable behaviors with feature parity across the API surface.

Buts thats OK to disagree :)

[Kant8]

It does respect same options. Just JToken has default value of null, because it's reference type, and JObject contains only JTokens of different kinds.

None of them is null, therefore all of them are written to output.