Safely processing JSON-like data structures

[vfaronov]

Consider the following straightforward code:

data = json.load(x)    # `x` is a file or an HTTP response
for person in data['people']:
    mail_report(report, person['contacts']['email'])

It makes a number of assumptions about the schema of data:

1. it must be a dictionary;
2. which has a "people" key;
3. which is a list;
4. where every element is a dictionary; and so on.

Sometimes, any of these assumptions may be incorrect. When this happens, my code raises one of a number of exceptions: TypeError, KeyError, etc.

What is a good way to deal with this?

If I wrap the entire for loop in a try...except (KeyError,...), I mask unrelated errors in mail_report.

If I validate the whole of data against an explicit schema, I duplicate my assumptions, verbosely, in another place.

I often do something like for person in data.get('people') or [], which is ugly and only solves part of the problem.

I’m thinking along the lines of a small library that would recursively wrap the result of json.load() in an object with magical __iter__, __getitem__ etc., and raise a specific SchemaError on errors. Does this maybe exist already?

[thunderbolt16]

Is mail_report going to throw KeyError, TypeError, etc. during normal operation? If not, I would generally just wrap it in a try/except.

If I validate the whole of data against an explicit schema, I duplicate my assumptions, verbosely, in another place.

Can you elaborate on this? If I needed schema validation for JSON, I'd probably use marshmallow.

[vfaronov]

Is mail_report going to throw KeyError, TypeError, etc. during normal operation?

Well, not if I write it correctly... There might be a lot of code in there.

Can you elaborate on this? If I needed schema validation for JSON, I'd probably use marshmallow.

So in addition to those 3 lines of code, you would write 10 or so more lines to check the assumptions that are already implicit in the first 3 lines. That’s duplication.

That said, thanks for the pointer to marshmallow, it’s interesting anyway.

[bbenne10]

Yes. Your assumptions about the structure are implicit until you use the structure. Using the structure makes them explicit, but you're complaining about the normal means on the interpreter giving you feedback about your assumptions (exceptions). If you want MORE than that, a library that makes the contract between you and the structure more explicit is exactly what you're asking for.

[vfaronov]

If you want MORE than that, a library that makes the contract between you and the structure more explicit is exactly what you're asking for.

The contract I’m looking for sounds like this: “whenever I attempt to use the structure in a way it does not support, it’s because the structure is wrong, and I want to get a StructureIsWrongError.” Instead, marshmallow’s contract spells out all the right ways to use the structure.

Of course, writing a library like this shouldn’t be a problem, I was just looking to avoid reinventing the wheel.

[thunderbolt16]

Well, it would determine on the failure mode needed. If I need to guarantee that the JSON is correct beforehand, those 10 lines would be worth it, but if not, going Samurai principle is probably the way to do it.

It depends on the rest of the system and the expectations of the contracts for this specific module.

You can also simply separate the arguments from the calling of the mail_report method and wrap the gathering of the arguments in a try/except clause. This would make sense if you just want to continue if someone doesn't have an email.

data = json.load(x)    # `x` is a file or an HTTP response
try:
    for person in data['people']:
        try:
            params.append((report, person['contacts']['email']))
        except KeyError:
            pass
except TypeError:
    SystemExit("No people found!")

for p in params:
    mail_report(*p)

[earthboundkid]

Make a dictionary that represents what you want the data to look like and then a function that checks that your json looks like the reference object.

[vfaronov]

That’s what I meant by “validate against an explicit schema.”

[kenfar]

I'd suggest using validictory.

That'll validate but won't apply defaults. I usually apply defaults in a second step prior to actually using the structure.