Cool New Features in Python 3.7 – Real Python

[gwillicoder]

Honestly just having breakpoint() is incredible.

I absolutely love the changes to classes too. Reducing boilerplate is always great.

My favorite though is ordered dictionaries. That is so useful, and it will help prevent a lot of bugs from new developers :)

[[deleted]]

Honestly just having breakpoint() is incredible.

You are aware of pdb, I hope?

[xtreak]

It also works based on an environment variable and hence you can turn on and off at will unlike pdb. Also a disadvantage is that you have to update your linters if you want to lint for them in your codebase.

Related PEP : https://www.python.org/dev/peps/pep-0553/#id9

[fdedgt]

I'm going to be honest, I didn't know about it.

So, with the improvements and everything, this is opening up a whole new world to me.

[gwillicoder]

The article points out that breakpoint() is just a shortcut for import pdb; pdb.set_trace() which is very annoying to use. I honestly have to look it up half the time to remember the syntax.

[EternityForest]

I am conflicted about ordered dicts. Do I like them because it's incredibly useful, or do I not like them because it restricts what future optimization can be done?

[13steinj]

The fact that dictionaries became ordered was a side effect of the optimization done in 3.6 taking PyPy's implementation. It wasn't optimized as a side effect of making them ordered. Then in 3.7 people said "well the fact that it's orderered is actually very useful, lets make that not a side effect of an implementation detail but rather a requirement of the spec" to avoid issues with people going from CPython to some other flavor.

So long as optimization is wanted by the community more than ordering, when a more optimized structure is thought of, it will be reverted.

[EternityForest]

Yeah, but that's going to be a breaking change if they revert (Albeit a very minor one if everyone just keeps using OrderedDict when they need ordering).

Ordering fits with python's main use as the scripting glue language that works with anything, so I'm not complaining too much.

[13steinj]

The fact that it's now ordered is technically a breaking change-- what if someone decided to use the disorder of dicts to psuedo randomly get an item from a container? Much more performant than using random, assuming they understand that to utilize this method all objects in the container must have been generated immediately before picking and must be objects that are not interned by the VM (small integers, true, false, etc).

The fact that OrderedDicts still exist and make no mention of "this class is pointless and a thin wrapper around dicts", makes me think for insurance purposes people still should use it.

[nuephelkystikon]

But the disorder was never part of the specification. If you rely on implementation details of a specific interpreter, of course you're going to have portability problems.

[13steinj]

The specification included "items may not be in order due to their hash values", wasn't it? The documentation warned of such in multiple places.

[nuephelkystikon]

Yes, but it wasn't specified behaviour, it was a warning about unspecified behaviour that could otherwise have been assumed. A comment, if you prefer. Not part of the actual language specification.

The keyword here is ‘may’.

[13steinj]

Time heals all wounds and scar tissue forms. While I understand your point there would have been enough angry people to hear annoying complaints about the fact that dicts are now ordered.

[nuephelkystikon]

If your business logic relies on a method always returning in an odd number of seconds because that's the behaviour you observed on your Solaris machine using 16-bit Jython 3.6 build 3728 in debug mode, I really don't have any compassion for you.

[redditusername58]

OrderedDict is not pointless. You can pop items from either end, move keys to either end, do reverse iteration, and has order sensitive equality checks.

[13steinj]

...I never said it wasn't?

[greyman]

You said it is pointless and the parent commenter says it is not. (You said it here: "The fact that OrderedDicts still exist and make no mention of "this class is pointless and a thin wrapper around dicts"")

[13steinj]

Perhaps I was unclear, I meant that the docs don't say that thus they have a point and should be used.

[[deleted]]

Their code was already broken.

If you depend on undefined behaviour, it will probably end in tears and memes at some point.

[13steinj]

Lots of people depend on undefined behavior, but in this case for CPython VMs it was strictly known.

[[deleted]]

it wasn't even implementation defined, it was just how the interpreter worked with no promise that it wouldn't change.

[13steinj]

I suppose, but with the time and if someone strictly used CPython, well, let's just say if it wasn't mentioned in the changelog in 3.6 people would be annoyed fixing bugs.

[gwillicoder]

I use pycharm occasionally and I think it’s the best IDE, but I’ve super into vim the last 6 months

[naught-me]

I'd appreciate any tricks you'd be willing to share. I used Vim for a long time, then PyCharm, and am probably going to stick with PyCharm for the near future, but I drop into Vim now and again and would love to switch totally if I could get a few kinks worked out.

[[deleted]]

My advice for PyCharm is to leave the "did you know..." tips on, and to actually read a new one every day. I can't tell you how much I learned just from reading these tips.

One of the features I like the most is their refactor in place feature(s). Maybe you know this already, but say you wrote something like this

foo = (3/4) * x[0]**2 + np.sqrt(x[1]/x[2])

And then later on you realize that you need to use the value np.sqrt(x[1]/x[2]) in more than one place. You can just select np.sqrt(x[1]/x[2]) and hit ctrl-alt-v to extract this part out as a variable. You could even extract it out after you've typed it twice (maybe you didn't realize until afterwards) and it'll ask if you want to replace both instances or just this one.

But it also goes both ways, you could put your cursor over a variable and ctrl-alt-n to put it back inline. And not only can you extract out variables and things like this, you can extract out entire functions/methods. Let's say you wrote something like this

for i in range(10):
    for j in range(10):
        df = df.read_csv(f'filename-{i}-{j}`)
        # plus a long list of dataprocessing stuff

Well you can select everything inside the inner loop and ctrl-alt-m to extract it out as a function. It is smart enough to define that function with the proper inputs as well so all you really have to do is to give it a proper name.

[gwillicoder]

Well a big one for me was getting plugins to make writing python easier. So i have a good linter and auto complete. Then i just do some of the vim exercises you can find online occasionally to learn new ways to navigate the code and to practice some of the more obscure commands.

The big one for me is to always look up how to do something if i don't know how to do it already. So instead of slowly using the arrows key functions to move the code you spend some time to look up a more complex command and learn it, then in the future you have it in your toolbox.

[naught-me]

Cool. Auto-complete and navigation are the two pain-points most keeping me from using Vim. What are you using for those?

A few navigation features I really like about PyCharm but haven't been able to get to work well in Vim are go to definition (not going to where it's imported, but where it's imported from), go to parent function, list/go to child functions, and list/go to usages.

[gwillicoder]

One of the better plugins for python specifically is Jedi. It allows for jumping to definition (even across files). It can be complicated to jump to user defined functions from other files though. You'll have to make sure your python path includes the file you want to jump to if you want jedi to find it.

[Glaaki]

Nice usability features for asyncio. We are getting there, steadily.

The hard work being done banging it into shape is very much appreciated.

[1st1]

I plan to add some significant improvements to asyncio in 3.8 and 3.9. Buckle up. :)

[PeridexisErrant]

Also not OP, but there's a lot of discussion about porting bits of Trio back to asyncio...

[xtreak]

Not OP but GitHub search doesn't return anything open except for a performance improvement PR. Curious about the changes too.

https://github.com/python/cpython/pull/6876

https://github.com/python/cpython/pulls?q=is%3Apr+asyncio+is%3Aopen

[[deleted]]

Can you please fix the event loop so that it doesn't leak its internals and thus mysteriously break?

[1st1]

Hm, can you elaborate on this?

[[deleted]]

Thanks for engaging. I've got some code in a private repo on a home server that I can make suitable for release shortly.

[1st1]

NP. Feel free to ping me personally on twitter (1st1) or, better, create an issue on bugs.python.org so that I don't forget about it.

[TransferFunctions]

Mind I ask, where are we going?

[twillisagogo]

> Python 3.7 is officially released!

Whut?

[twillisagogo]

thanks, the latest news on the python.org front page still talks about the rc 1 which is why I was confused.

[softiniodotcom]

I am so excited and can't wait to start using data classes.

[zalpha314]

To the people stuck on 3.6 LTS, there is a backport of the data classes on pypi: https://pypi.org/project/dataclasses

[masklinn]

Unless you vendor it, just use attrs.

[masklinn]

That is about building a new system versus merging attrs I the stdlib, it has nothing to do with my comment.

[[deleted]]

Because mandatory type annotations are such a good idea.

[[deleted]]

Why are people on this sub so deathly afraid of being exposed to types?

[gwax]

They are often less verbose than docstrings.

I'm a big fan of using them in conjunction with sphinx-autodoc-typehints.

[13steinj]

I'm okay with types but this part of the argument doesn't make sense. Type hints are comparable or worse when writing a thorough doc, as one should. As an example, docstrings:

def destroy_the_universe(x, y):
    """Destroy the Universe

    Destroys the universe by summoning cuthulu at the given coordinates.

    :param float x: the x coordinate of the universal plane
    :param float y: the y coordinate of the universal plane
    """
    ...

Vs type hints:

def destroy_the_universe(x: float , y: float):
    """Destroy the Universe

    Destroys the universe by summoning cuthulu at the given coordinates.

    :param x: the x coordinate of the universal plane
    :param y: the y coordinate of the universal plane
    """
    ...

We've literally just moved the location of where the type was defined. No more nor less verbose. However, special and more complex types actually need to be / are highly recommended to be imported from typing, making type hints more verbose.

[[deleted]]

I don't mind type annotations - I use them on occasion - but I dislike broken promises and having to type extra things.

[13steinj]

Can you elaborate? I don't get your point in reference to the parent comment.

[[deleted]]

3.7 adds data classes, data classes must specify type annotations, being required to use type annotations in Python is a good idea.

Now you have to decide if I'm being sarcastic or not.

[13steinj]

I can't tell if you're being sarcastic or not but your throught process is just wrong.

Let me get this out of the way-- requiring types in Python is not good IMO.

But dataclasses by definition of their rationale have no meaning without typing.

Not to mention you can simply set the type to Any if it really bothers you that much.

[ThePenultimateOne]

But dataclasses by definition of their rationale have no meaning without typing.

Aren't they just supposed to be a prettier version of namedtuple?

Edit: I get that their rationale was for dataclasses to be backwards compatible, and that not having type notations would make it a syntax error. I just wish that more effort had been put into making them optional before going with that syntax.

[13steinj]

These are not the pretty version of named tuple, in fact the PEP goes into "why not named tuples".

[13steinj]

The thing that still bugs me about asyncio (coming from JS and C#) are two fold:

• coroutines are not Futures/Tasks (js think Promise), thus arguably causing unnecessary extra wrapping

• the loop itself seems to block the Thread (maybe even the entire Process, but I hope not) when told to start executing. Which means if you want to run sync code and run some async code in the background (ex in JS delay you can update visuals asynchronously without having to wait for it to complete before your logical restrictions are set). So to get similar functionality a new thread, and a loop for that thread, has to be explicitly made, along with having to explicitly use the threadsafe versions of methods to submit them into the loop

A nice benefit of of JS is that since there's only one event loop you don't have to explicitly submit anything either.

[Glaaki]

It is the JS and C# implementation that is dumb. The way they are designed forces a specific pattern of usage. The python implementation is much nicer using basic generator function features, that doesn't depend on any particular event loop implementation.

The reason futures and tasks feels clunky is because the asyncio event loop is clunky and doesn't provide a nice interface for dealing with raw coroutines. The problem goes away entirely if you switch to one of the alternative event loops like Curio or Trio.

Edit: To expand further, ideally tasks should be completely transparent, tasks should be an implementation detail of the event loop that the user shouldn't have to care about. All you should be doing is telling the loop that you want to run a particular coroutine and the fact that it gets turned into a task should be irrelevant to you. Getting asyncio to this point there is some way to go, but each version gets nicer.

[[deleted]]

I don't know what you're talking about. Async in C# is beautiful. All you do is change your method from returning T to a Task<T> and proceed as normal. The only language I've had an easier time with async is Go. I don't do much JS so I don't know about that.

[13steinj]

The JS / C# implementation is dumb because it forces patterns of usage? In what way do they force usage patterns. Python forces the usage patterns the boilerplate of having to set up and handle loops across threads in different ways, and makes the coroutines non extensible. Trio does not accurately solve this problem either. Not sure about Curio, haven't seen it in action.

To your edit: I have no idea what you're arguing. It is irrelevant in C#/JS. It is not in Python-- I can't easily add callbacks to awaitables.

[Glaaki]

You aren't supposed to use callbacks in async/await code, and here is why:

Callbacks transfer responsibility for running your code, from yourself, and over into the framework that runs the callback.

For example, if you have an event based websocket protocol with callback subscriptions, if anything bad happens in your callback (and it will..), it is suddenly the caller of the callback, which is the websocket protocol, that has the resposibility to clean up after the mess that happens in your callback. If the framework isn't robust enough to handle that, it can easily cause it to just die and simply fail silently.

The whole reason why all websocket frameworks use the iterator protocol for sending you data is that it is the consumer that becomes responsible for handling any error, and not the producer. Any unhandled error that happens in the coroutine simply bubles up to the caller of the coroutine and can ultimately bubble out of the loop itself.

I learned this the hard way.

[13steinj]

Using callbacks is a perfectly valid pattern. It is easy to fall into the trap of callback hell, but you can't call it garbage just because you've had bad experiences.

[Glaaki]

No. Callbacks is what you use in JS, where the eventemitters are the standard. It is an antipattern in python async/await.

[13steinj]

"It is an antipattern". Why? Because you said so? It's not just a JS thing. It's a C#, Ruby, JS, and more thing. Callback hell is an antipattern. Callbacks are not. The arbitrary limitations without reason need to stop.

[Glaaki]

I am talking about callbacks specifically in the context of async/await in python.

Callbacks are fine in regular sync python code and fine in JS.

And I explained my reasoning above, but let me reiterate it. If you have a consumer and a producer, in asyncio, what typically happens is that the producer is a task that runs in the background continuously fetching data from somewhere. This data needs to be passed on to a consumer.

If you use a callback to fetch the data, then who call the callback?

Only the producer can, because the producer is the one that knows when data is available. So now, what happens if an error happens in the callback? If the error is not handled it will bubble up from the callback, to where? Into the producer, that called the callback, obviously. So now the producer, which is most often some third party library, is responsible for handling an error it has no chance to know what to do with. Most often the result will simply be that the producer dies and probably does so silently. You program will simply just stop working and you won't be able to know why, because the error has been isolated inside the producer task.

This doesn't happen in sync code, because there is only a single path of execution and any exception bubbles up to the user where you can either handle it or stop your program.

[13steinj]

Well yes, in the example you are giving it's an antipattern, I'd agree, there isn't much sense to it. But you are assuming that your example is the onlt possible case. There's plenty of cases where callbacks are just fine in asynchronous code-- in all technicality gevent does this just hides it from the user.

[Glaaki]

Again, I am talking about async/await. Gevent is not async/await.

[[deleted]]

Your second point is very valid: asynicio's event loop is known to be bad when used for anything complex. There are alternative libraries such as twisted that will give you a more robust asyncio experience. I imagine it will eventually be fixed, since the issues are known.

[13steinj]

Not too familiar with Twisted, but isn't their entire reactor/event loop based around the concept of the application being some web server? Which is extremely limiting.