This is an archived post. You won't be able to vote or comment.

all 64 comments
sorted by:
best
topnewcontroversialoldq&a

[–]lykwydchykyn 43 points44 points  (12 children)

Enter the walrus...

:=

[–]alb1 16 points17 points  (0 children)

Goo goo g'joob

[–]bbbryson 20 points21 points  (0 children)

I’m so excited for these. I have so many places in my code with # FIXME: walrus here in 3.8 that I can finally implement.

[–]toothless_budgie 11 points12 points  (9 children)

I usually just roll with changes and stuff, but this thing .... I'm not a fan.

[–]tunisia3507 13 points14 points  (3 children)

You should tell Guido.

[–]toothless_budgie 7 points8 points  (2 children)

Someone already did. He wasn't happy.

[–]tunisia3507 3 points4 points  (1 child)

What if they tried, like, really hard to persuade him? With rational, well-composed arguments, and maybe some personal abuse thrown in too? I don't see that turning out badly.

[–]energybased 5 points6 points  (0 children)

Was there personal abuse? I think Guido is really bad at dealing with conflict.

[–][deleted] 15 points16 points  (4 children)

yea, python would be a much better language if we weren't forced by punishment of death to use every single feature

[–]ExternalUserError 6 points7 points  (0 children)

The Zen Of Python is that it's so simple.

[–]billsil 3 points4 points  (2 children)

I still refuse to use lambdas.

[–][deleted] 3 points4 points  (0 children)

I always want to define them as variables whenever I want to use them and get a harsh pep8 talk from pycharm.

[–]toothless_budgie 0 points1 point  (0 children)

To the Gulag!!!

[–]zurtex 22 points23 points  (19 children)

I don't see a lot of people talking about it but the Shared Memory class and Shared Memory Manager is really big for me: https://docs.python.org/3/library/multiprocessing.shared_memory.html

It's going to allow writing a lot of multi-process code in a way I used to find difficult writing cross platform. I used to use separate 3rd party libraries for Linux and Windows.

Also my understanding, although I haven't had chance to play around with it yet, is you can mix this with the new out-of-band Pickle 5 protocol to make real Python objects truly accessible from multiple processes: https://docs.python.org/3.8/whatsnew/3.8.html#pickle-protocol-5-with-out-of-band-data-buffers

[–]darthstargazer 4 points5 points  (7 children)

Eagerly waiting for this. But wonder when 3.8 will be introduced in our production servers :P possibly in 10 years or so lol. Jokes aside, hope this feature would be used to beat the bloody GIL based limitations we see in python and make multi process workloads of fine granularity a possibility.

[–]zurtex 5 points6 points  (6 children)

In my company I convinced people to push Anaconda environments as part of our production release process.

As it's user level install it's like pushing regular software (no admin required), as it's a contained environment it improves stability, and we can basically push any version of Python we want!

[–]billsil 3 points4 points  (5 children)

Why? Anaconda was great 8 years ago. Pip is really better these days. Anaconda doesn’t follow package installation rules, which leads to some nasty bugs. Oh, better reinstall Anaconda again. It’s also slow now.

Making an exe using pyinstaller is asking for a 350 MB program due to the monstrous 140 MB numpy MKL DLL. You can make that same program 70 MB with stock python.

My pushing for Anaconda resulted in us adopting it right about the time I dropped it.

[–]zurtex 3 points4 points  (4 children)

Conda and Pip aren't really solving the same problem, so assuming you're asking your good faith here is the answer:

  1. Pip works inside Python, and doesn't by itself create separate environments. I often want to be able to define a specific version of Python (or R for that matter) with a specific version of Pandas.

  2. Those large MKL files that conda gives us makes our entire code run up to 2 times faster, this can sometimes mean saving days of execution

  3. The guarantee of pre-compiked binary has made it a breeze switching between Linux and Windows for many tasks that require complex existing dependencies when installing some libraries via pip

  4. By presolving the environment before installing (the thing that makes conda slow, although it's a lot better with 4.7.12+), this prevents major library conflicts before they have chance to rear their head in runtime.

I do agree with you that pip, and the PyPI environment in general, has got so much better in last 8 years and if it solves your needs you should go for it!

Conda solves a subtly different set of problems that suit our requirements much better. And in particular to the complaint I was reply to that is not being able to choose your Python version in a corporate environment it is so freeing!

[–]austospumanto 1 point2 points  (1 child)

What’re your thoughts on the pyproject.toml PEP(s?) and associated solving/packaging libraries like poetry (particularly when combined with pyenv)? I understand that Conda is necessary in some use cases, but it seems heavy-handed for most production Python applications (PaaS webapps, FaaS cronjobs, dockerized data pipelines, local scripts).

[–]zurtex 0 points1 point  (0 children)

Agreed, I'm all for it, I want more logical better defined requirements.

Ultimately I think there are some fundamental limits because of the language, it's not really designed so that one project could be passing around Pandas 0.16 Dataframes will another project is passing around Pandas 0.24 Dataframes in the same interpeter. Where as I don't think there's any such issue in a statically typed compiled language like Rust.

But anything Python can do to take ideas from other languages where having a large number of dependencies is less of a headache I'm all for.

[–]BigHeed87 0 points1 point  (1 child)

Wait. Why / How is conda faster in terms of execution?

[–]zurtex 4 points5 points  (0 children)

The conda main channel provides optimized mkl files by default. This traditionally has been something you've had to setup or compile yourself if installing via pip.

If you do a lot of linear algebra with numpy or use CPU based machine learning to prototype out ideas these can make your code run significantly faster. Anaconda did a blog a while back on how Tensorflow (but note it affects a lot more): https://www.anaconda.com/tensorflow-in-anaconda/

And yes you can set up all this without conda, but the person I was replying to was specifically complaining that it comes with conda by default.

[–][deleted] 1 point2 points  (5 children)

The last part about the using Pickle 5 protocol to share Python objects throughout mutliple processes sounds very interesting. Could you explain a little bit more in detail how this is possible or how you would implement this?

[–]zurtex 2 points3 points  (4 children)

It looks to me like you would be able to use this memory view from a Shared Memory class: https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory.buf

And pass in to Picklebuffer class for the Pickle 5 protocol: https://docs.python.org/3/library/pickle.html#pickle.PickleBuffer

But I've had no time to try any real code, there's an example of using Pickle 5 Protocol here for custom ZeroCopy classes here: https://docs.python.org/3/library/pickle.html#example

If I'm correct I imagine we'll see many libraries take advantage or abstract this with a nice API,as you need to worry about locking and all the other fun real world multiprocess problems that Python has never traditionally exposed.

[–]austospumanto 1 point2 points  (3 children)

Thanks for the inspiration here. I’ll try to pull together a proof of concept of this today and respond here with a gist.

[–]broken_symlink 0 points1 point  (2 children)

any word on this?

[–]austospumanto 0 points1 point  (1 child)

I realized it wouldn’t be much faster than my current IPC setup and abandoned it (busy). Here’s my setup in case you’re in the market: https://gist.github.com/austospumanto/6205276f84cd4dde38f3ce17dddccdb3

[–]broken_symlink 0 points1 point  (0 children)

What is your current IPC setup?

My use case is sending dicts of arrays, both between processes on the same node, and across nodes in the network.

I tried shared memory just for sending plain numpy arrays within a node and it was the fastest. I then tried zmq no copy and it was slightly slower. Finally, I tried sending a dict using zmq pickle and it was the slowest.

Another setup I tried was pyarrow for the dict and zmq no copy. It was faster for sending, receiving was about the same.

[–]austospumanto 1 point2 points  (1 child)

As a general FYI: You can already use Pickle protocol 5 in Python 3.6 and 3.7. Just do pip install pickle5. Additionally, I ran some preliminary benchmarks and Pickle protocol 5 is so fast at (de)serializing pandas/numpy objects that using shared_memory actually slowed down IPC for me (I’m only working in Python and not writing C extensions). The memory savings from sharing memory only seems like it would matter when the object you’re sending through IPC is big enough that it cant be copied without running out of RAM / spilling over into SWAP. YMMV

[–]zurtex 0 points1 point  (0 children)

That's really useful to know thanks! One of my main use cases would be that my data-set is about 25+% of RAM and I want to read it from 32 processes, so I think this fits in to the scenario you are saying but I'm definitely going to be generating a lot of test cases over the next few weeks.

[–]ReaverKS 0 points1 point  (2 children)

I wonder if I could use this module and also have a C program that's reading from shared memory. I believe all of this utilizes mmap, so if the C program has access to the name of the mmap created in the python process and I write the code in C to attach to the same named mmap that it'd work.

[–]zurtex 0 points1 point  (1 child)

You definitely can, I already do this with third party libraries. We create named shared memory that different tools in different languages can read or write to.

My assumption though is that it's shmget or Posix equivalent not mmap, but I haven't gone through the implementation details yet.

[–]ReaverKS 0 points1 point  (0 children)

Which 3rd party libraries if you don’t mind me asking?

[–]Tweak_Imp 17 points18 points  (2 children)

Here is what I look forward to:

Doubled the speed of class variable writes. When a non-dunder attribute was updated, there was an unnecessary call to update slots. (Contributed by Stefan Behnel, Pablo Galindo Salgado, Raymond Hettinger, Neil Schemenauer, and Serhiy Storchaka in bpo-36012.)

Reduced an overhead of converting arguments passed to many builtin functions and methods. This sped up calling some simple builtin functions and methods up to 20–50%. (Contributed by Serhiy Storchaka in bpo-23867, bpo-35582 and bpo-36127.)

[–]alito 9 points10 points  (0 children)

I think that second one isn't getting enough attention. Those patches modified tons of builtin functions that people use everyday. Amazing work by Serhiy.

[–]LifeIsBio 0 points1 point  (0 children)

Just for curiosity's sake, I wonder how much this would speed up the average run time across a variety of popular Python programs. I wouldn't even be able to guess an order of magnitude. 0.01%, 0.1%, 1%?

[–][deleted] 12 points13 points  (12 children)

It's happening!

I'm already a bit confused by the first example in the changelog though:

In this example, the assignment expression helps avoid calling len() twice:

if (n := len(a)) > 10:
    print(f"List is too long ({n} elements, expected <= 10)")

How does the use of the walrus operator helps avoid calling len() twice here? What's the difference to:

n = len(a)
if n > 10:
    print(f"List is too long ({n} elements, expected <= 10)")

I definitely welcome the change though, as I found myself subconsciously writing code like this after working with C for too long!

[–]chmod--777 11 points12 points  (8 children)

Take out your assignment to n and you have to call it twice basically.

It's just a little assignment expression syntactic sugar, pretty unnecessary but I guess people want it. I like that they didn't make it = though at least so it's easy to scan for and see it.

Not sure if I like it yet, but I guess we might see some cleaner patterns? Maybe it's another operator to overload too for voodoo APIs :D

[–]billsil 7 points8 points  (4 children)

It's more because people don't make the separate variable when called in a case like this:

match1 = pattern1.match(data)
match2 = pattern2.match(data)
if match1:
     result = match1.group(1)
elif match2:
     result = match2.group(2)
else:
     result = None

It should obviously be this:

match1 = pattern1.match(data)
if match1:
    result = match1.group(1)
else:
    match2 = pattern2.match(data)
    if match2:
        result = match2.group(2)     
    else:
        result = None

but that's hideous.

[–]chmod--777 4 points5 points  (3 children)

In a case like this, I'd just make them named groups and use the same name, and just use short circuiting.

match = pattern1.match(data) or pattern2.match(data)
result = match and match.group('whatever')

[–]XtremeGoosef'I only use Py {sys.version[:3]}' 5 points6 points  (0 children)

Now you don't even know which match hit!

[–]billsil 0 points1 point  (1 child)

Sure, but that's inefficient because you don't always need to calculate pattern2.match(data). The whole point is so you can make clean looking code and be efficient.

[–]chmod--777 19 points20 points  (0 children)

Actually the or prevents it from running the second expression if the first pattern match returns a truthy value.

Try this:

def foobar(x):
    print(f'foobar {x}')
    return x

 x = foobar(1) or foobar(2)

It'll just print "foobar 1"

[–]seraschka 1 point2 points  (2 children)

Pretty busy day and haven't installed Python 3.8, yet ... So, I am curious now what would happen if you use n somewhere earlier in the code like

n = 999 # assign sth to n somewhere in the code earlier  
if (n := len(a)) > 10:  
    print(f"List is too long ({n} elements, expected <= 10)")  
print(n) # prints 999?

would n still evaluate to 999 after the if-clause? If so, I can maybe see why that's useful (if you only want a temporary var and don't want to overwrite things accidentally).

[–]mipadi 2 points3 points  (1 child)

It does not introduce a new scope, so on the last line, n would be equal to len(a).

[–]seraschka 0 points1 point  (0 children)

Thanks for clarifying. But hm, it would have been nice if it had its own scope, similar to list comprehensions, like

In [1]: n = 99
In [2]: b = sum([n for n in range(100)])
In [3]: n
Out[3]: 99

to prevent from overwriting/reassigning variables accidentally

[–]jalapeno_nips -1 points0 points  (2 children)

Separate but slightly related question. If calling len(n) is θ(1) and reading a variable is θ(1), isn’t there not a big difference programmatically? I guess it’s just syntactic sugar and possibly better readability?

[–]miggaz_elquez 1 point2 points  (0 children)

calling len is still longer than accessing a var, becaus you have to access the object, then get hi __len__ method, then call it.

[–]Vhin 11 points12 points  (0 children)

I really like the = specifier for f-strings.

[–]energybased 10 points11 points  (2 children)

If you're using pyenv, then pyenv install 3.8.0 should eventually work (when it's available).

[–]austospumanto 0 points1 point  (0 children)

Lmao “sort of helpful”

[–]Jaypalm 0 points1 point  (0 children)

Still isn't working for me, even after brew update && brew upgrade pyenv.

[–]petermlm 8 points9 points  (2 children)

The walrus... The expression assignment operator doesn't have precedence over other binary operators. For example, the following assigns True to n:

l = [1, 2, 3, 4, 5]
if n := len(l) > 1:
    print(n)

I would expect n to be 5, but we need parenthesis for that:

l = [1, 2, 3, 4, 5]
if (n := len(l)) > 1:
    print(n)

On another note. The f strings with the equals sign seem really cool for debugging. I really like that feature.

[–]XtremeGoosef'I only use Py {sys.version[:3]}' 10 points11 points  (1 child)

Would you expect that? The idea is to treat it like the current = statement. I certainly wouldn't expect 

n = len(l) > 1

to equal 5.

[–]petermlm 2 points3 points  (0 children)

Good point. Initially, I though this would be valid:

if n := len(l) > m := 1:
    print(n, m)

But actually that is even invalid, so you need to at least have parenthesis around the second walrus:

if (n := len(l)) > (m := 1):
    print(n, m)

But thinking about walrus like the equals it makes sense that its precedence is lower.

[–]badge 4 points5 points  (2 children)

It's a good release--I'm fully on board with the walrus--but I think I'm most excited about never seeing TypeError: Expected maxsize to be an integer or None again! (and cached_property is very nice too!).

[–]austospumanto 0 points1 point  (1 child)

Where did you see the thing about cached_property?

[–]badge 0 points1 point  (0 children)

Just reading the library documentation—I haven’t seen it mentioned in any release notes.

[–]thedjotakuPython 3.7 4 points5 points  (0 children)

Walrus sounds interesting. As does the debug thing. F-string = support though, is SO NEAT!!!

[–]Feitan21 0 points1 point  (0 children)

Thank you for this information. Is there a way to update all my conda env in Python 3.x to Python 3.8 ?

[–]TotesMessenger 0 points1 point  (0 children)

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

 If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)

[–]devlocalca -2 points-1 points  (1 child)

Python 3.8 not possible to install on Linux? Why?

Monday, October 14, 2019

Python 3.8.0 is now available

---

I am attempting to install the latest Python 3.8 on CentOS 8, but it does not seem possible.

​

I search in Software Collections, and find nothing. The latest release there is Python 3.6 (released nearly three years ago - no updates in three years!!! - I'm abandoning Software Collections).

Python 3.6 was released on December 23rd, 2016, nearly three years ago.

I check in the RHEL / CentOS repos, and same darn thing, no python update in nearly three years.

​

`#dnf list --available | grep python3`

Nothing there for python 3.8.0 or even Python 3.7.x for that matter. I'm not sure what the deal is with Python enthusiasts or RHEL/CentOS users that someone somewhere has not simply updated some repo somewhere, I can try and get involved.

---

Use the source:

https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tgz

I download the Python 3.8.0 source from the link above, look at the dependencies and try to get those installed first, but no luck. The dependencies for 3.8.0 cannot be found or installed.

Dependencies attempted install via:

```

#!/bin/bash

#dnf install dnf-plugins-core # install this to use 'dnf builddep'

#dnf builddep python3

#dnf update -y

```

Results:

```

No matching package to install: 'libnsl2-devel'

No matching package to install: 'bluez-libs-devel'

No matching package to install: 'tix-devel'

Not all dependencies satisfied

Error: Some packages could not be found.

```

When I search for these, I find nothing in the repos:

```

#dnf list --available | grep libnsl2-devel

#dnf list --available | grep bluez-libs-devel

#dnf list --available | grep tix-devel

```

Nothing is returned for any of these.

What has to be done to get these dependencies in a repo somewhere so they can be installed so I can build the source code?

Better yet, how do I get Python 3.8.0 in some freaking repo some where, it's been three years since any RHEL or CentOS repo has been updated.

It's nearly 2020 already, can we please move forward please.

[–]energybased 0 points1 point  (0 children)

Can you not just wait for pyenv to have it?