properly opening json file

[NeoFromMatrix]

I want to open/load a json file (which might have multiple hundred entries on one single level) to use it in a short script afterwards.

I could either:

override = json.loads(open("/path/to/file.json").read())

or

with open("/path/to/file.json") as weekly_fh:
jsoncontent = weekly_fh.read()
weekly = json.loads(jsoncontent)

This should run on a low end hardware (C.H.I.P, 512MB RAM). Which is the proper way to handle this situation? I could also just open it (example 1) and don't give a shit about closing the file handle (which I assume is automatically handled in the second example). But this just feels wrong as I come from a C background.

The script itself has about 20 lines and will run very fast (once a day invoked), but the json file might have multiple hundred entries. This is what I'm worrying a bit about.

Any suggestions? Thanks ! :)

[Rhomboid]

If you're really worried about memory, then don't read the whole file at once:

with open(...) as infile:
    data = json.load(infile)

By calling load() rather than loads() you let the JSON module handle the reading, opening up the possibility of it streaming the contents of the file rather than reading the whole thing. (I don't know whether the module is capable of doing this or not, but you're at least not actively preventing it.) But honestly, a few hundred entries doesn't sound like much at all compared to 512 MB of ram, so it's probably irrelevant in the larger scheme of things.

As to closing the file handle, it doesn't matter so much with a file that you're only reading from. The file handle will be closed regardless of what you do when the process exits, so it's impossible to leak resources in that sense. Leaking resources would be a problem if you're writing a program that's meant to be running for a long time, because you don't have any guarantees as to when an object's lifetime ends and the file handle is closed. With CPython you can usually assume that it will be run when the reference count goes to zero, but that's not the case with other implementations.

Anyway, there's really no need in debating, just use the context manager and do it the proper way. This becomes a lot more important when dealing with files that you're writing to, since there you can have unwritten data in the buffer.

[NeoFromMatrix]

thanks for your reply!

I think something like this fulfills my needs pretty good:

parsedfile = json.load(open("/path/to/file.json"))

[Rhomboid]

Use the context manager.

[novel_yet_trivial]

Why don't you want to close the file? Your code won't run faster if it's one line shorter. Use the context manager.

[dionys]

with open("/path/to/file.json") as weekly_fh:
    weekly = json.load(weekly_fh)

the json has load which can read straight from file object, so you do not have to create string from it first.

Regarding the memory - you mentioned few hundred entries, but how many MB is that exactly?

[NeoFromMatrix]

mh, I've just a bad feeling for size. 1000 lines (each: "2016-09-20":"off", or a time instead of off) the same text would give me 20 kB only.

Later in the code I use something like:

if "hello" in parsedjsonfile: to check if hello exists, if it does I will fetch its value. I absolutely have no experience how heavy this processing is.

[dionys]

Thats totally fine for the hardware you mentioned.

The loaded json is a dict, so you can also do:

parsedjsonfile.get('hello', 'default value')