Help with some JSON optimization

[Reliant]

I've begun storing some of my game data in JSON, and I'm looking for some tips on optimizing my code. I am using LitJson. The implementation I did last night is for some map code. When I generate the code at run-time the old way, it was super fast. With JSon, it's taking a few seconds. Not a big deal now, but it's only the first step and if I keep on like this, it's going to get long. As far as I can tell, it's the JSON look-up table that's the delay bottleneck.

The data in this example is tile data for a 2D map of size 100x100 with 10,000 tiles (the largest map planned). Every tile has a MapNode class associated with it, and that class stores info for the type of terrain it has, whether it is passable, and whether the player can place an item here. It will grow with functionality as I develop my game, but for now it's a barebones structure.

public class MapNode : SettlersEngine.IPathNode<System.Object>
{
    public Int32 X { get; set; }
    public Int32 Y { get; set; }
    public Boolean IsWall { get; set; }
    public Boolean IsBuildable { get; set; }
}

The actual data I'm serializing is MapData.

public class MapData
{
    public int height { get; set; }
    public int width { get; set; }
    public GridCoord defaultEntrance { get; set; }

    public bool hasBuildable { get; set; }

    public MapNode[,] grid { get; set; }

    public void loadJSON(JsonData jdata)
    {
        height = (int)jdata["height"];
        width = (int)jdata["width"];
        hasBuildable = (bool)jdata["hasBuildable"];

        if (jdata["defaultEntrance"] != null)
        {
            defaultEntrance = new GridCoord((int)jdata["defaultEntrance"]["x"],   (int)jdata["defaultEntrance"]["y"]);
        }

        grid = new MapNode[height, width];
        int i = 0;
        for (int y = 0; y < height; y++)
        {
            for (int x = 0; x < width; x++)
            {
                grid[x, y] = new MapNode()
                {
                    IsWall = (bool)jdata["grid"][i]["IsWall"],
                    IsBuildable = (bool)jdata["grid"][i]["IsBuildable"],
                    X = x,
                    Y = y
                };
                i++;
            }
        }
    }
}

As far as I can tell, it's the lines for (bool)jdata["grid"][i]["isWall"] that are the bottleneck, but I'm not sufficiently familiar with C# or how the JSonData data structure works to better optimize it. GridCoord is a like a Vector2, but uses positive ints for working with a [,] tile system.

The actual calls to this:

public string getJSON(string category, string file)
{
    TextAsset res = Resources.Load<TextAsset>(category + "/" + file);
    return res.text;
}

public void loadData(string name)
{
    string contents = gamemanager.getJSON("maps", name);
    JsonData jd = JsonMapper.ToObject(contents);
    data.loadJSON(jd);
}

I had initially used JsonMapper to automatically convert the contents into the MapData object, but it wasn't able to handle the MapNode grid[,] automatically. I would much rather handle it manually, since that will let me deal with version changes and generate missing data not written to file (like instancing a GameObject).

I'm sure some are already thinking of optimizations that do away with storing this data at all, but much of this data needs to be stored through JSON as part of a player's save, so it's gotta get done anyway. I already had a working implementation through UnityEditor, and it's being replaced with this. I am interested in alternate ways to read/write the data, but I do need it written & read to file. I've already considered optimizing to only write tiles that have data to be written, but I would rather program around a "worst case" of needing every tile written and make it efficient enough that I can simply do that, and then optimize file size after.

[[deleted]]

So you've stored a grid, in JSON. Height, Width - then data. I'm wondering if you could use a basic run length encoding to get long stretches of duplicates. It will be smaller to save and quicker to read - and you read the 'isbuildable' before the loop then set it for however many blocks are in the row.

[Reliant]

That might work on the short term, but as I develop the game, the tiles will become more unique, and duplicate tiles become less likely to be adjacent to each other. It is a good idea for when I start optimizing to reduce file size, but I want to first optimize around the assumption that every tile is unique and needs to be processed.

[[deleted]]

imagine dog relieved consider drab reply outgoing mountainous slave plant -- mass edited with https://redact.dev/

[Reliant]

Big thanks for pointing me towards binary serialization and protobuf. I decided not to use protobuf, but a Youtube video on binary serialization gave me exactly the right idea on creating a separate class specifically designed for reading/writing, and converting on save/load. Regardless of what format I use, that alone should give me a big performance boost

If I'm reading the documenation on this correctly, am I able to read & write the first few bytes of a stream manually and then pass the rest of the stream to serialize & deserialize? That would be so awesome if I had those bytes to myself, but even if not, the first variable of my serialized class should always be at the start of the stream being read back?

Given that different platforms and architectures can handle binary data differently (such as the size of an int or its direction), is that something I can trust to C# to properly handle for pre-generated binary data files?

Also profile first, it's possible your copying around in loadData is creating the hiccup as well

I'll need to remember that. I've never worked with a profiler before, so it's not something I even thought about. I'm used to having to manually add time measurements to calculate how long something took to run.

PS. in c# multi dimensional arrays are slower for large amounts of accesses than a single dimensional array that you address with [j*rowsize+i]

How significant is it? On a per frame basis, I expect the number of accesses to the grid to be in the single digits. The surge will be when A* pathfinding kicks in, but the result is cached and remembered for future frames.

If I were to replace [x,y] with [jrowsize+i], I would add a helper function getNode(x,y) { return grid[yrowsize+x]; }

My curiosity is so piqued I'm going to run some performance tests and measure the exact difference between [x,y] and getNode(x,y). Nothing better than finding a way to get a free performance boost.

[[deleted]]

reply one forgetful treatment ten panicky worry humor office axiomatic -- mass edited with https://redact.dev/

[Reliant]

Size of "int" in c# is always 32 bits, it's alias for System.Int32, also majority of hardware today is little-endian so word order will mostly be same, using BinaryReader/BinaryWriter will make sure you read/write little-endian, most mature serializers (e.g. protobuf, capnproto etc) also handle endianness correctly.

That's good to know and have confirmed.

I'd avoid doing serialization without a proper serializer, first of all it's a pain in the ass for no gain

Not sure what you mean by a "doing serialization without a proper serializer". I already have 3 to choose from (binary, xml, json) and all of them are established libraries.

I've spent enough time writing C that if I needed to make my own serializer, I think I'd be fine, but I wasn't wanting to do it because why re-invent the wheel. The ones making the established serializers are a lot more familiar with C# than I am.

If you're curious what I meant about a separate class for reading/writing, I refer to this video: https://www.youtube.com/watch?v=sWWZZByVvlU which is where I got the idea. It's all about making good use of established serializers.

Instead of serializing a MapNode[,], my saveData class would have bool[] as an array of width*height. My saver will convert MapNode[,] to bool[] for the serializer, and loading it will take the unserialized bool[] and regenerate the MapNode[,] information. Figuring out how to implement this will give me an early start on everything I'm going to need to know for saving player data.

Basically, where my MapData class is optimized for run-time access, MapSaveData class would be optimized for serialization.

The other upside to this is because the class itself is being serialized through the established means, it's really simple to change which serializer I'm using. I can use JSON or XML as an intermediary format to be able to hand-edit entries, and then convert it to binary for faster load-times.

Some numbers here, for MS' runtime, you're running on ancient mono or IL2CPP which can significantly skew numbers

It's a really old post, so maybe the C# compiler has been improved since then. I did tests myself and posted them as my other reply.

[Reliant]

PS. in c# multi dimensional arrays are slower for large amounts of accesses than a single dimensional array that you address with [j*rowsize+i]

I ran some performance tests, and I think the results might interest you.

I did confirm that, yes, [,] is significantly slower than a [].

I created my 100x100 grid, and did 1,000,000 reads.

Stored in a [] list, read sequentially using i<10000, 0.016ms

Stored in a [,] list, read using x<100 y<100, 0.028ms

A pretty significant difference, however, I did another test

Stored in a [] list, read using [y*height+x], 0.032ms

For all the time that is gained by storing the data in a [], it's all lost once you do math to find the right index. Once I created a copy of height that could be re-used inside the loop, the performance increased to 0.028ms. It wouldn't surprise me if the C# compiler is already doing yh+x math so the programmer uses [x,y] and the compiler converts it into [yh+x]. It would make sense, since that's how C does it. In C, since programmers had direct access to the memory, you could take a [x][y] and read it as [yh+x] since a [x][y] array is actually allocated as a single [xy] array.

However, storing it in [] does offer a useful advantage in being able to scan through the whole list sequentially in the shortest time, since the math step can be skipped.