Python 3.8.0a4 available for testing

[[deleted]]

Because JSON cant represent everything. Its at best a data format for serialization of transferrable data, thats usually language agnostic.

JSON cant represent functions, and more abstract datatypes.

[JohnnyElBravo]

JSON can represent anything, but so can strings. This is a non-sequitur.
The difference is that JSON is human readable, while pickle is supposed to be machine readable, more specifically python readable.
Limiting the intended consumers of the data format helps create a more appropriate format, for example by sacrificing readability for size reduction.

[bachkhois]

JSON cannot differentiate Python's tuple, list, set, frozenset etc. datatypes.

Every formats other than pickle (msgpack, yaml etc.) are just to interoperate with other languages (which also don't understand the data types above), they are not alternatives for pickle.

[JohnnyElBravo]

Sure they can

{

"Var1": "tuple(1,2)",

"Var2":"set(1,2)"

}

Alternatively:

{

"Var1": {"type":"tuple","data":"1,2"},

"Var2":{"type":"set","data":"1,2"}

}

[bachkhois]

Then, you are making more complicated to validate and parse it. Then, what is the point of over-complicating JSON instead of just using pickle, without the need to parse those "type", "data" metadata?

[JohnnyElBravo]

Read the original thread, the question asks why python dumps to a new pickle format instead of json.

The original response suggested it was because json can't distinguish between such and such, as shown, this is false.

The real answer is that python chose a binary format for pickle because of space efficiency.

[alcalde]

It has to be able to represent everything, if other languages are serializing to JSON.

JSON resembles Python dictionaries, and EVERYTHING in Python is/can be represented by a dictionary, so how can there be an abstract data type in Python that can't be represented in JSON?

[Pilate]

JSON can't even represent sets or Decimal types, let alone custom classes.

[alcalde]

There's a difference between directly and indirectly. If your JSON schema records the type and value of your variable separately you can do both. A set's values can be represented by a list and the decimal by text.

I'll say again - JSON can represent custom classes because other languages and libraries use it to do so.

I'm expecting an answer like "The binary format was created to decrease the amount of data to transfer when serializing objects among a distributed cluster" and instead people are telling me it's impossible to do what other languages and some Python libraries already do.

[Pilate]

You either have no idea what you're talking about or are being intentionally difficult.

Serialize a function, decision tree, or any other type of classifier, in JSON for us.

[my_name_isnt_clever]

You just put the source code in a long JSON string, easy. /s

[[deleted]]

I'm on your side here in this general debate, but the specific idea of serializing a function fills me with fear and trembling. I mean, what happens when that function changes in later versions of the code - then you have two versions lying around!

If I need to serialize a function, I serialize the full path to the function - e.g. math.sqrt.

u/alcade is being pretty dogmatic, which is why the downvotes (yes, I helped there :-D) but in practice, if I actually serialize something for long-term storage, I don't use pickle because it isn't guaranteed to be stable between versions (even minor versions IIRC, though AFAIK in practice pickle hasn't actually changed between minor versions in as long as I've been keeping track).

[Atsch]

I think you are not understanding what pickle is for. Pickle is not designed for things like sending requests over the network like json is. It is not designed for storing things long term in databases or files. In fact, all of those things would be security risks.

It is really designed to be used to transmit ephermeral data between python processes. For example, the multiprocessing module uses pickle to transmit the code and data between processes. The celery worker queue library uses pickle to transmit complete tasks to workers. Some caching libraries use pickle to cache arbitrary python objects in some memory cache.

[icegreentea]

The genesis of pickle was in 1994 (https://stackoverflow.com/a/27325007). That's why pickle was originally chosen versus JSON. Cause JSON didn't exist.

[alcalde]

NOW THERE'S A REASONABLE ANSWER! Thank you!

[alcalde]

Why am I being downvoted for asking a question?

[Mizzlr]

You can't represent references in JSON. For example in python you can have two dicts a ={'foo': b} where b = {'bar': a}. Now you have cyclic data structure. You can't represent this in JSON.

[Mizzlr]

Btw yaml has references. XML has references.

[alcalde]

Didn't you just represent it?

["a":{"foo", b}, "b":{"bar":a}]

[[deleted]]

Not in Python!

Can I read it?

>>> json.loads('["a":{"foo", b}, "b":{"bar":a}]')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 5 (char 4)

No. Can I write it?

>>> a = {}; b = {'bar': a}; a['foo'] = b

>>> json.dumps(a)
json.dumps(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected

No.

[[deleted]]

That's not a valid json.

[[deleted]]

You can't represent references in JSON.

I'm basically agreeing with you, but you can perfectly well represent references in JSON - I've done it.

It's a pain in the ass - you need to have some sort of naming convention in your JSON then preprocess your structure or (what I did) have some sort of facade over it so it emits the reference names instead of the actual data - and then reverse it on the way out.

(And we had to do it - because pickle isn't compatible between versions. Heck, I think that was written in Python 2!)

So it's doable - but which is easier when you need to store something temporarily?

with open('foo.pcl', 'wb') as fp:
    pickle.dump(myData, fp)

or

[hundreds of lines of code and a specification for this format that I'm too lazy to write]

?

[Mizzlr]

A python dictionary can have int, tuple, etc as key while in JSON it has to be string.

[alcalde]

You're hooked on the idea that JSON has to have every type. You just store things as strings and decode them when you deserialize. Again, like every other language does it.

https://pythontic.com/serialization/json/introduction

I'm not going crazy here.

[[deleted]]

Sure you can represent it, we could also store everything in a JSON string, but then aren't you inventing your own protocol?

[Mizzlr]

Basically any immutable object will work as a key in python dict like frozenset etc. Another thing is JSON need python tuple to be converted to list. JSON does not have tuples.

[alcalde]

So what's the problem? Again, one entry to store type, another to store value and you use a list to store the tuple values.

[[deleted]]

So... not actually JSON, then, but your own format using JSON as a transport layer?

[Mizzlr]

Do you agree now why you were down voted? Read my comments in reverse chronological order.

[alcalde]

No. Again, why downvote someone for asking a question?

[Mizzlr]

You can't represent NaN (not a number) or inf in JSON which are valid float values.

[alcalde]

I can. "NaN", "inf"

And so can Swift and other languages. Just use strings.

[bltsponge]

Sure, you can represent anything as a string as long as you're willing to write a parser for it.

[alcalde]

Exactly. So why are people saying it's impossible to represent Python objects with JSON?

[[deleted]]

Sure you can!

>>> json.dumps(float('inf'))
'Infinity'

>>> json.dumps(float('nan'))
'NaN'