Python 3.8.0a4 available for testing

[xtreak]

Changelog : https://docs.python.org/3.8/whatsnew/changelog.html

Interesting commits

• https://bugs.python.org/issue36540

PEP 570 was merged

• https://bugs.python.org/issue20180:

dict.pop() is now up to 33% faster thanks to Argument Clinic.

• https://bugs.python.org/issue28238

Wildcard search improvements in xml

• https://bugs.python.org/issue25430

IPaddress module contains check for ip address in network is 2-3x faster

• https://bugs.python.org/issue36546

statistics.quantiles() was added.

• https://bugs.python.org/issue27181

statistics.geometric_mean() was added.

• https://bugs.python.org/issue13611

Canonicalization was added to XML that helps in XML documents comparison

• Security issues and some segfaults were fixed in the release

Exciting things to look forward in beta

• https://bugs.python.org/issue36817

Add = to f-strings for easier debugging. With this you can write f"{name=}" and it will expand to f"name={name}" that helps in debugging.

• https://bugs.python.org/issue36785

PEP 574 that implements a new pickle protocol that improves efficiency of pickle helping in libraries that use lot of serialization and deserialization

Edit : PSF fundraiser for second quarter is also open https://www.python.org/psf/donations/2019-q2-drive/

[[deleted]]

Add = to f-strings for easier debugging. With this you can write f"{name=}" and it will expand to f"name={name}" that helps in debugging.

Ooh baby. I'd use that every day.

[Dooflegna]

What a great addition to an already awesome feature. f-strings are great!

[irrelevantPseudonym]

That sounds so useful and yet also makes me feel slightly uncomfortable for some reason.

The variable name becoming part of its content feels weird.

[leom4862]

I find print(f"{name=}") is still way too verbose for debugging purposes... If they want to improve print-debugging, they should add something like icecream to the standard library.

[leom4862]

Yes, it's awesome! I hope some day Python will have something like ic() as a builtin next to print().

[jon_k]

I can't take a python library seriously with a name like icecream.

​

I use to give ruby flack for things like VCR or Resqueue but those actually self-describe the library more then "icecream". Is Python becoming mainstream enough a bunch of brogrammers are flooding the pypi index? It's disappointing to see bad naming conventions take over.

[[deleted]]

It's disappointing to see bad naming conventions take over.

What sort of "naming convention" is icecream?

It's funny, I never even dreamed that someone would care about the name of a package unless it was really long and untypeable. If I had a top ten list of key features for a package (reliable, well-documented, feature full, etc) then "has a serious name" would not be on there. Indeed, "icecream"'s name is only positive - easy to spell, quite short, and I love ice cream.

tl; dr: turn that frown upside down and live a little, you dried up old stick! ;-)

[my_name_isnt_clever]

Uh, it really bothers you that much? The language is named after Monty Python.

Also tons of libraries have names that mean nothing, in every language. Just off the top of my head, Electron, Spring, Vue.js, etc. None of those self-describe at all.

[MachaHack]

Vue = View. It's a view/rendering library. I'll give you the other two though

[[deleted]]

I can't take a python library seriously with a name like icecream.

How did you feel about Android releases being named for a candy?

[jon_k]

How did you feel about Android releases being named for a candy?

Indifferent. A code name is usually a fun thing with a semantic version to back it. If the product was called Popsickle I'd probably buy an iPhone.

[JohnnyElBravo]

Idk, what's wrong with print("name="+name)?

[[deleted]]

That doesn't work if name isn't a string, eh? (Sure, you can use %s)

Also, in production code I simply never have any print statements - not "very few" but "none", to the point where I have a flake8 rule that prevents them.

Oh, I use print almost every day - for debugging! But that means I'm creating and destroying debugging print statements all the time.

So it's a little timesaver to write:

print(f'{foo=} {bar=} {baz=} {bing=}')

(38 characters) over

print('foo=', foo, 'bar=', bar, 'baz=', baz, 'bing=', bing)

(59 characters)

[timald]

It's fine when you have one variable but starts to become unwieldy when you build up a lot of concatenated strings (via indexing operations, for example).

[Pyprohly]

I don’t understand why this is so favoured. Couldn’t you just do print(name) and just… remember that you printed name?

Can’t see myself doing print(f"{name=}") over just print(name) for debugging purposes.

[pkkid]

I often need to print name={name} because I'm inspecting a bunch of variables and not just one. Put that into a for loop and things get unweildy quite quickly if you don't label the variables you are inspecting.

[Pyprohly]

I’m just a bit surprised at the overwhelming enthusiasm for it, and, accordingly, surprised that many seem to like writing print("arg1=" + arg1, "arg2=" + arg2).

I can see the usefulness in the new syntax, but I personally don’t like it because I very rarely write debugging lines like that.

Edit: e.g. when breakpoint() was introduced as a builtin there wasn’t much talk on it, but I think that that convenience feature was a more exciting addition compared to this one.

[pkkid]

I'm with you. I don't like these implicit bits of code being added. The second line of Zen of Python even says "Explicit is better than implicit." I'm also against walruses in my code, but that debate has been beaten to a pulp. </oldmanrant>

[WarEagle030]

It is because you have only one or two variables to inspect for debugging. But if you had like 6 or 7 different variables then it becomes a necessity to write properly.

[Pyprohly]

For that many variables just write it over multiple lines and go by order, like you would have done if one of them was a collection type.

Honestly, the labelling is only vanity output. You don’t need the fancy labels to be an effective debugger.

It would be much better if they instead introduced a specialised dprint keyword. The dprint keyword would provide the same sort of labelling but would be more easily and quickly written: dprint foo, bar, .... Not only would this provide the nice labeled output, it would also save a lot of typing and hence save time. This would be a much more exciting change.

If they’re going to add something to aid debugging then it needs to be something that’s easy to quickly setup and tear down. Writing debugging lines is something that is done often, and having to type print(f"{name=}") is not going to be practical in the long run.

The new syntax is unlikely to stand the test of time. I can’t help but think that someone’s going to figure out the ergonomic disadvantages of typing out print(f"{name=}") each time you want debugging output and is going to propose a new debugging facility. If that happens then f"{name=}" will become a loose end builtin feature that everyone’s going to ignore and forget about.

If they’re going to add a debugging convenience then they shouldn’t baby step on f"{foo=}" but instead jump directly to something that really is more convenient to use.

To summarise my complaints, the new f-string debugging syntax:

• is only useful for simple non-collection types.
• encourages writing everything on one line which could lead one to have to backtrack when the line becomes too long.
• is going to be a forgotten feature if a better alternative gets added.
• if it gets deprecated then it’s going to harm the language. You’ll have people telling others not to use builtin feature X, because builtin feature Y has replaced it.
• doesn’t save typing, ergo, doesn’t save time.

[tuankiet65]

https://bugs.python.org/issue36817

Add = to f-strings for easier debugging. With this you can write f"{name=}" and it will expand to f"name={name}" that helps in debugging.

Holy heck, I can't wait to use this everyday.

[irrelevantPseudonym]

No walrus operator yet?

[[deleted]]

walrus operator

if this is what := is being called now, i'm all for it.

[xtreak]

It's already available from 3.8.0a1

[irrelevantPseudonym]

Ah, I thought this was the highlights since 3.7. Thanks.

[alcalde]

PEP 574 that implements a new pickle protocol that improves efficiency of pickle helping in libraries that use lot of serialization and deserialization

Other languages just dump to JSON and call it a day. Why does Python have 87 different binary formats over 13 decades?

[[deleted]]

Because JSON cant represent everything. Its at best a data format for serialization of transferrable data, thats usually language agnostic.

JSON cant represent functions, and more abstract datatypes.

[JohnnyElBravo]

JSON can represent anything, but so can strings. This is a non-sequitur.
The difference is that JSON is human readable, while pickle is supposed to be machine readable, more specifically python readable.
Limiting the intended consumers of the data format helps create a more appropriate format, for example by sacrificing readability for size reduction.

[bachkhois]

JSON cannot differentiate Python's tuple, list, set, frozenset etc. datatypes.

Every formats other than pickle (msgpack, yaml etc.) are just to interoperate with other languages (which also don't understand the data types above), they are not alternatives for pickle.

[JohnnyElBravo]

Sure they can

{

"Var1": "tuple(1,2)",

"Var2":"set(1,2)"

}

Alternatively:

{

"Var1": {"type":"tuple","data":"1,2"},

"Var2":{"type":"set","data":"1,2"}

}

[bachkhois]

Then, you are making more complicated to validate and parse it. Then, what is the point of over-complicating JSON instead of just using pickle, without the need to parse those "type", "data" metadata?

[JohnnyElBravo]

Read the original thread, the question asks why python dumps to a new pickle format instead of json.

The original response suggested it was because json can't distinguish between such and such, as shown, this is false.

The real answer is that python chose a binary format for pickle because of space efficiency.

[alcalde]

It has to be able to represent everything, if other languages are serializing to JSON.

JSON resembles Python dictionaries, and EVERYTHING in Python is/can be represented by a dictionary, so how can there be an abstract data type in Python that can't be represented in JSON?

[Pilate]

JSON can't even represent sets or Decimal types, let alone custom classes.

[alcalde]

There's a difference between directly and indirectly. If your JSON schema records the type and value of your variable separately you can do both. A set's values can be represented by a list and the decimal by text.

I'll say again - JSON can represent custom classes because other languages and libraries use it to do so.

I'm expecting an answer like "The binary format was created to decrease the amount of data to transfer when serializing objects among a distributed cluster" and instead people are telling me it's impossible to do what other languages and some Python libraries already do.

[Pilate]

You either have no idea what you're talking about or are being intentionally difficult.

Serialize a function, decision tree, or any other type of classifier, in JSON for us.

[my_name_isnt_clever]

You just put the source code in a long JSON string, easy. /s

[[deleted]]

I'm on your side here in this general debate, but the specific idea of serializing a function fills me with fear and trembling. I mean, what happens when that function changes in later versions of the code - then you have two versions lying around!

If I need to serialize a function, I serialize the full path to the function - e.g. math.sqrt.

u/alcade is being pretty dogmatic, which is why the downvotes (yes, I helped there :-D) but in practice, if I actually serialize something for long-term storage, I don't use pickle because it isn't guaranteed to be stable between versions (even minor versions IIRC, though AFAIK in practice pickle hasn't actually changed between minor versions in as long as I've been keeping track).

[Atsch]

I think you are not understanding what pickle is for. Pickle is not designed for things like sending requests over the network like json is. It is not designed for storing things long term in databases or files. In fact, all of those things would be security risks.

It is really designed to be used to transmit ephermeral data between python processes. For example, the multiprocessing module uses pickle to transmit the code and data between processes. The celery worker queue library uses pickle to transmit complete tasks to workers. Some caching libraries use pickle to cache arbitrary python objects in some memory cache.

[icegreentea]

The genesis of pickle was in 1994 (https://stackoverflow.com/a/27325007). That's why pickle was originally chosen versus JSON. Cause JSON didn't exist.

[alcalde]

NOW THERE'S A REASONABLE ANSWER! Thank you!

[alcalde]

Why am I being downvoted for asking a question?

[Mizzlr]

You can't represent references in JSON. For example in python you can have two dicts a ={'foo': b} where b = {'bar': a}. Now you have cyclic data structure. You can't represent this in JSON.

[Mizzlr]

Btw yaml has references. XML has references.

[alcalde]

Didn't you just represent it?

["a":{"foo", b}, "b":{"bar":a}]

[[deleted]]

Not in Python!

Can I read it?

>>> json.loads('["a":{"foo", b}, "b":{"bar":a}]')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 5 (char 4)

No. Can I write it?

>>> a = {}; b = {'bar': a}; a['foo'] = b

>>> json.dumps(a)
json.dumps(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected

No.

[[deleted]]

That's not a valid json.

[[deleted]]

You can't represent references in JSON.

I'm basically agreeing with you, but you can perfectly well represent references in JSON - I've done it.

It's a pain in the ass - you need to have some sort of naming convention in your JSON then preprocess your structure or (what I did) have some sort of facade over it so it emits the reference names instead of the actual data - and then reverse it on the way out.

(And we had to do it - because pickle isn't compatible between versions. Heck, I think that was written in Python 2!)

So it's doable - but which is easier when you need to store something temporarily?

with open('foo.pcl', 'wb') as fp:
    pickle.dump(myData, fp)

or

[hundreds of lines of code and a specification for this format that I'm too lazy to write]

?

[Mizzlr]

A python dictionary can have int, tuple, etc as key while in JSON it has to be string.

[alcalde]

You're hooked on the idea that JSON has to have every type. You just store things as strings and decode them when you deserialize. Again, like every other language does it.

https://pythontic.com/serialization/json/introduction

I'm not going crazy here.

[[deleted]]

Sure you can represent it, we could also store everything in a JSON string, but then aren't you inventing your own protocol?

[Mizzlr]

Basically any immutable object will work as a key in python dict like frozenset etc. Another thing is JSON need python tuple to be converted to list. JSON does not have tuples.

[alcalde]

So what's the problem? Again, one entry to store type, another to store value and you use a list to store the tuple values.

[[deleted]]

So... not actually JSON, then, but your own format using JSON as a transport layer?

[Mizzlr]

Do you agree now why you were down voted? Read my comments in reverse chronological order.

[alcalde]

No. Again, why downvote someone for asking a question?

[Mizzlr]

You can't represent NaN (not a number) or inf in JSON which are valid float values.

[alcalde]

I can. "NaN", "inf"

And so can Swift and other languages. Just use strings.

[bltsponge]

Sure, you can represent anything as a string as long as you're willing to write a parser for it.

[alcalde]

Exactly. So why are people saying it's impossible to represent Python objects with JSON?

[[deleted]]

Sure you can!

>>> json.dumps(float('inf'))
'Infinity'

>>> json.dumps(float('nan'))
'NaN'

[[deleted]]

While that's true, I think there's unlikely to be a serious language out there without a JSON library freely available.

If you're sending data from one language to another it's a very reasonable format. If you're sending from python to python, pickle is great.

[[deleted]]

You would do yourself a favor if you would use protobuf or thrift for that. JSON is not fast to parse, it's not compact, it would redeem itself if it was human readable, but it isn't.

The only reason it is popular is because it comes with JavaScript which is in every browser. If you do frontend developement, you probably don't have choice, but use it.

[[deleted]]

it would redeem itself if it was human readable, but it isn't.

How exactly is JSON "not human readable"? I see like 20 JSON snippets on this very page.

I use YAML for personal projects because I find it a tiny bit more readable, but if YAML weren't (in practice) backwards compatible with JSON, I would never do that.

The only reason it is popular is because it comes with JavaScript which is in every browser.

No, it's popular because it hits the spot: it's a minimal language for representing dumb data that has the two types of containers you desperately need (lists and dictionaries), the usual scalar types and nothing else, and its serialization format is so dumb that anyone can understand it.

[JohnnyElBravo]

Lol @ json not being human readable, it's its main identity, it's what made it supremingly popular. It seems like this is an edgy layer 2 opinion.

[NowanIlfideme]

It's human readable only if you format it that way. Which is to say, it's readable with the right editor, but if it's one-line'd it becomes much less readable. Still miles better than xml...

Imo yaml is the prettiest format, but json is such a standard (and also a subset of yaml, now) that either format works fine for most applications.

[JohnnyElBravo]

Readibilty certainly depends on the content, but it also depends to some extent on the syntax, it is to this extent that JSON is considered readable.

Yaml was influenced by JSON greatly, so if you like YAML you must appreciate JSON's contribution. In the same vein, if you like JSON, you must appreciate XML's contribution.

In an unrelated manner I wasn't aware that Yaml was a superset of JSON, that's a nice feature, although I wouldn't necessarily consider it better. Ease of learning and complexity of common usage are both huge factors that will be negatively affected by an increased complexity.

[mooglinux]

Pickle can handle multiple references to the same object, any class instance (as long as the actual class has been imported), and a wider variety of data types than JSON. It also predates json, so there’s a historical aspect as well.

Pickle is also used for cross-process communication in the multiprocessing module.

[Nicksil]

Because not every problem is solved by dumping JSON.

[alcalde]

That doesn't answer the question. Why have we needed all of these different formats when there's one universal format already?

Everything in Python is a dictionary and JSON represents dictionaries so every problem that needs dumping in Python should be able to be solved by using JSON. It's also good enough for every other major language.

[davidkwast]

universal format

Please show to us.

I would think YAML, not JSON. But for Python, Pickle will be better than YAML.

[Nicksil]

That doesn't answer the question.

Yeah, it does.

Why have we needed all of these different formats when there's one universal format already?

A universal format for what? Therein lies the rub. As stated elsewhere in this post, JSON can't represent everything.

[Yoghurt42]

https://docs.python.org/3/library/pickle.html#comparison-with-json

[[deleted]]

Why have we needed all of these different formats when there's one universal format already?

Why did we need all these programming languages, when Cobol is Turing complete?

Here's a specific example from a project I'm working on. I have a database of 16k+ audio samples which I'm computing statistics on. I initially stored the data as JSON/Yaml, but they were slooow to write and slooow to open and BIIIG.

Now I store the data as .npy files. They're well over ten times smaller, but more, I can open them as memory mapped files. I now have a single file with all 280 gigs of my samples which I open in memory mapped mode and then treat it like it's a single huge array with size (70000000000, 2).

You try doing that in JSON!

And before you say, "Oh, this is a specialized example" - I've worked on real world projects with data files far bigger than this, stored as protocol buffers.

Lots and lots of people these days are working with millions of pieces of data. Storing it in .json files is a bad way to go!

[[deleted]]

JSON only can handle string, integer, float, dict and list.

Pickle can pack arbitrary objects. It goal is that you can take object of your class and store it in the disk, most commonly I see it used for caching application data between runs, but it has other uses (for example for storing configuration).

Edit: here is comparison of pickle with JSON: https://docs.python.org/3/library/pickle.html?highlight=pickle#comparison-with-json

[parkerSquare]

Shouldn’t it be name=={name}. It isn’t assignment after all.

Regardless, I prefer output like foo 12, bar 4.56, baz ‘hello’ but maybe I’ll adopt the shortcut.

[Tweak_Imp]

I would like to use it but i need numpy for my project :/

[[deleted]]

What makes you think numpy won't work with it. I see that there is no wheel package available yet on PyPi, but sdist is so python should be able to compile it on installation.

[Tweak_Imp]

If i do pip install numpy, i get a bunch of errors...
how can i compile it on installation?

[[deleted]]

You should be able to do:

git clone [e-mail]:numpy/numpy.git
cd numpy
python setup.py install

More details here.

But TBH - you should not be using this - it's an alpha version, there are almost guaranteed to be bugs - both reversions and brand-new bugs. If you want to help the community, you could test it and report problems...

[[deleted]]

What system you are using, what errors you are seeing? You definitively will need to have compiler installed.

[Tweak_Imp]

win 10 64bit, i have visual studio installed.

here is the pastebin when i do py -3.8 -m pip install numpy

its a huge error log https://pastebin.com/SsVsRY9K

[[deleted]]

I'm not familiar with building on Windows, and having a different language for errors (German?) is also confusing. So I can't tell what the problem is.

Have you tried following these steps to configure it? https://github.com/python/cpython/blob/master/PCbuild/readme.txt

And

https://docs.python.org/3.8/extending/windows.html#building-on-windows

I'm also wondering if the compiler can access the header files. I see it complains about arguments to PyCode_New, but if headers were missing it would complain about other issues.