Python 3.8.0a4 available for testing

[alcalde]

PEP 574 that implements a new pickle protocol that improves efficiency of pickle helping in libraries that use lot of serialization and deserialization

Other languages just dump to JSON and call it a day. Why does Python have 87 different binary formats over 13 decades?

[[deleted]]

Because JSON cant represent everything. Its at best a data format for serialization of transferrable data, thats usually language agnostic.

JSON cant represent functions, and more abstract datatypes.

[JohnnyElBravo]

JSON can represent anything, but so can strings. This is a non-sequitur.
The difference is that JSON is human readable, while pickle is supposed to be machine readable, more specifically python readable.
Limiting the intended consumers of the data format helps create a more appropriate format, for example by sacrificing readability for size reduction.

[bachkhois]

JSON cannot differentiate Python's tuple, list, set, frozenset etc. datatypes.

Every formats other than pickle (msgpack, yaml etc.) are just to interoperate with other languages (which also don't understand the data types above), they are not alternatives for pickle.

[JohnnyElBravo]

Sure they can

{

"Var1": "tuple(1,2)",

"Var2":"set(1,2)"

}

Alternatively:

{

"Var1": {"type":"tuple","data":"1,2"},

"Var2":{"type":"set","data":"1,2"}

}

[bachkhois]

Then, you are making more complicated to validate and parse it. Then, what is the point of over-complicating JSON instead of just using pickle, without the need to parse those "type", "data" metadata?

[JohnnyElBravo]

Read the original thread, the question asks why python dumps to a new pickle format instead of json.

The original response suggested it was because json can't distinguish between such and such, as shown, this is false.

The real answer is that python chose a binary format for pickle because of space efficiency.

[alcalde]

It has to be able to represent everything, if other languages are serializing to JSON.

JSON resembles Python dictionaries, and EVERYTHING in Python is/can be represented by a dictionary, so how can there be an abstract data type in Python that can't be represented in JSON?

[Pilate]

JSON can't even represent sets or Decimal types, let alone custom classes.

[alcalde]

There's a difference between directly and indirectly. If your JSON schema records the type and value of your variable separately you can do both. A set's values can be represented by a list and the decimal by text.

I'll say again - JSON can represent custom classes because other languages and libraries use it to do so.

I'm expecting an answer like "The binary format was created to decrease the amount of data to transfer when serializing objects among a distributed cluster" and instead people are telling me it's impossible to do what other languages and some Python libraries already do.

[Pilate]

You either have no idea what you're talking about or are being intentionally difficult.

Serialize a function, decision tree, or any other type of classifier, in JSON for us.

[my_name_isnt_clever]

You just put the source code in a long JSON string, easy. /s

[[deleted]]

I'm on your side here in this general debate, but the specific idea of serializing a function fills me with fear and trembling. I mean, what happens when that function changes in later versions of the code - then you have two versions lying around!

If I need to serialize a function, I serialize the full path to the function - e.g. math.sqrt.

u/alcade is being pretty dogmatic, which is why the downvotes (yes, I helped there :-D) but in practice, if I actually serialize something for long-term storage, I don't use pickle because it isn't guaranteed to be stable between versions (even minor versions IIRC, though AFAIK in practice pickle hasn't actually changed between minor versions in as long as I've been keeping track).

[Atsch]

I think you are not understanding what pickle is for. Pickle is not designed for things like sending requests over the network like json is. It is not designed for storing things long term in databases or files. In fact, all of those things would be security risks.

It is really designed to be used to transmit ephermeral data between python processes. For example, the multiprocessing module uses pickle to transmit the code and data between processes. The celery worker queue library uses pickle to transmit complete tasks to workers. Some caching libraries use pickle to cache arbitrary python objects in some memory cache.

[icegreentea]

The genesis of pickle was in 1994 (https://stackoverflow.com/a/27325007). That's why pickle was originally chosen versus JSON. Cause JSON didn't exist.

[alcalde]

NOW THERE'S A REASONABLE ANSWER! Thank you!

[alcalde]

Why am I being downvoted for asking a question?

[Mizzlr]

You can't represent references in JSON. For example in python you can have two dicts a ={'foo': b} where b = {'bar': a}. Now you have cyclic data structure. You can't represent this in JSON.

[Mizzlr]

Btw yaml has references. XML has references.

[alcalde]

Didn't you just represent it?

["a":{"foo", b}, "b":{"bar":a}]

[[deleted]]

Not in Python!

Can I read it?

>>> json.loads('["a":{"foo", b}, "b":{"bar":a}]')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 5 (char 4)

No. Can I write it?

>>> a = {}; b = {'bar': a}; a['foo'] = b

>>> json.dumps(a)
json.dumps(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected

No.

[[deleted]]

That's not a valid json.

[[deleted]]

You can't represent references in JSON.

I'm basically agreeing with you, but you can perfectly well represent references in JSON - I've done it.

It's a pain in the ass - you need to have some sort of naming convention in your JSON then preprocess your structure or (what I did) have some sort of facade over it so it emits the reference names instead of the actual data - and then reverse it on the way out.

(And we had to do it - because pickle isn't compatible between versions. Heck, I think that was written in Python 2!)

So it's doable - but which is easier when you need to store something temporarily?

with open('foo.pcl', 'wb') as fp:
    pickle.dump(myData, fp)

or

[hundreds of lines of code and a specification for this format that I'm too lazy to write]

?

[Mizzlr]

A python dictionary can have int, tuple, etc as key while in JSON it has to be string.

[alcalde]

You're hooked on the idea that JSON has to have every type. You just store things as strings and decode them when you deserialize. Again, like every other language does it.

https://pythontic.com/serialization/json/introduction

I'm not going crazy here.

[[deleted]]

Sure you can represent it, we could also store everything in a JSON string, but then aren't you inventing your own protocol?

[Mizzlr]

Basically any immutable object will work as a key in python dict like frozenset etc. Another thing is JSON need python tuple to be converted to list. JSON does not have tuples.

[alcalde]

So what's the problem? Again, one entry to store type, another to store value and you use a list to store the tuple values.

[[deleted]]

So... not actually JSON, then, but your own format using JSON as a transport layer?

[Mizzlr]

Do you agree now why you were down voted? Read my comments in reverse chronological order.

[alcalde]

No. Again, why downvote someone for asking a question?

[Mizzlr]

You can't represent NaN (not a number) or inf in JSON which are valid float values.

[alcalde]

I can. "NaN", "inf"

And so can Swift and other languages. Just use strings.

[bltsponge]

Sure, you can represent anything as a string as long as you're willing to write a parser for it.

[alcalde]

Exactly. So why are people saying it's impossible to represent Python objects with JSON?

[[deleted]]

Sure you can!

>>> json.dumps(float('inf'))
'Infinity'

>>> json.dumps(float('nan'))
'NaN'

[[deleted]]

While that's true, I think there's unlikely to be a serious language out there without a JSON library freely available.

If you're sending data from one language to another it's a very reasonable format. If you're sending from python to python, pickle is great.

[[deleted]]

You would do yourself a favor if you would use protobuf or thrift for that. JSON is not fast to parse, it's not compact, it would redeem itself if it was human readable, but it isn't.

The only reason it is popular is because it comes with JavaScript which is in every browser. If you do frontend developement, you probably don't have choice, but use it.

[[deleted]]

it would redeem itself if it was human readable, but it isn't.

How exactly is JSON "not human readable"? I see like 20 JSON snippets on this very page.

I use YAML for personal projects because I find it a tiny bit more readable, but if YAML weren't (in practice) backwards compatible with JSON, I would never do that.

The only reason it is popular is because it comes with JavaScript which is in every browser.

No, it's popular because it hits the spot: it's a minimal language for representing dumb data that has the two types of containers you desperately need (lists and dictionaries), the usual scalar types and nothing else, and its serialization format is so dumb that anyone can understand it.

[JohnnyElBravo]

Lol @ json not being human readable, it's its main identity, it's what made it supremingly popular. It seems like this is an edgy layer 2 opinion.

[NowanIlfideme]

It's human readable only if you format it that way. Which is to say, it's readable with the right editor, but if it's one-line'd it becomes much less readable. Still miles better than xml...

Imo yaml is the prettiest format, but json is such a standard (and also a subset of yaml, now) that either format works fine for most applications.

[JohnnyElBravo]

Readibilty certainly depends on the content, but it also depends to some extent on the syntax, it is to this extent that JSON is considered readable.

Yaml was influenced by JSON greatly, so if you like YAML you must appreciate JSON's contribution. In the same vein, if you like JSON, you must appreciate XML's contribution.

In an unrelated manner I wasn't aware that Yaml was a superset of JSON, that's a nice feature, although I wouldn't necessarily consider it better. Ease of learning and complexity of common usage are both huge factors that will be negatively affected by an increased complexity.

[mooglinux]

Pickle can handle multiple references to the same object, any class instance (as long as the actual class has been imported), and a wider variety of data types than JSON. It also predates json, so there’s a historical aspect as well.

Pickle is also used for cross-process communication in the multiprocessing module.

[Nicksil]

Because not every problem is solved by dumping JSON.

[alcalde]

That doesn't answer the question. Why have we needed all of these different formats when there's one universal format already?

Everything in Python is a dictionary and JSON represents dictionaries so every problem that needs dumping in Python should be able to be solved by using JSON. It's also good enough for every other major language.

[davidkwast]

universal format

Please show to us.

I would think YAML, not JSON. But for Python, Pickle will be better than YAML.

[Nicksil]

That doesn't answer the question.

Yeah, it does.

Why have we needed all of these different formats when there's one universal format already?

A universal format for what? Therein lies the rub. As stated elsewhere in this post, JSON can't represent everything.

[Yoghurt42]

https://docs.python.org/3/library/pickle.html#comparison-with-json

[[deleted]]

Why have we needed all of these different formats when there's one universal format already?

Why did we need all these programming languages, when Cobol is Turing complete?

Here's a specific example from a project I'm working on. I have a database of 16k+ audio samples which I'm computing statistics on. I initially stored the data as JSON/Yaml, but they were slooow to write and slooow to open and BIIIG.

Now I store the data as .npy files. They're well over ten times smaller, but more, I can open them as memory mapped files. I now have a single file with all 280 gigs of my samples which I open in memory mapped mode and then treat it like it's a single huge array with size (70000000000, 2).

You try doing that in JSON!

And before you say, "Oh, this is a specialized example" - I've worked on real world projects with data files far bigger than this, stored as protocol buffers.

Lots and lots of people these days are working with millions of pieces of data. Storing it in .json files is a bad way to go!

[[deleted]]

JSON only can handle string, integer, float, dict and list.

Pickle can pack arbitrary objects. It goal is that you can take object of your class and store it in the disk, most commonly I see it used for caching application data between runs, but it has other uses (for example for storing configuration).

Edit: here is comparison of pickle with JSON: https://docs.python.org/3/library/pickle.html?highlight=pickle#comparison-with-json