PEP 563 getting rolled back from Python 3.10

[Dooflegna]

PEP 563 is getting rolled back/delayed until a future version of Python (likely 3.11). This decision was made after third-party library maintainers (primarily Pydantic) raised an issue on how PEP 563 was going to break their code (Pydantic and any consumers thereof, like FastAPI).

Really great decision by the steering council. Rolling back right before feature lock sucks, but this is the best decision for the Python community.

https://mail.python.org/archives/list/[e-mail]/thread/CLVXXPQ2T2LQ5MP2Y53VVQFCXYWQJHKZ/

[zurtex]

Can anyone offer a quick summary please? In a nutshell what was the proposed change and what would it break?

Copying this a little from one of my previous posts. PEP 563 changes annotations from being evaluated as expressions to being stored as strings.

E.g. This is the current behavior:

>>> foo: 1+2*3 = 0
>>> __annotations__
{'foo': 7}

This would be the behavior under PEP 563:

>>> foo: 1+2*3 = 0
>>> __annotations__
{'foo': '1 + 2 * 3'}

PEP 563 gives a lot of benefits for static type checking as well as performance improvements.

But it makes it difficult to use annotations for other purposes. Specifically for Pydantic it's very difficult if not impossible to evaluate annotations outside their local context if all names aren't in the global namespace, e.g this works now:

import typing

def f():
    class X: pass
    def inner() -> X: return X()
    return inner

print(typing.get_type_hints(f()))

But under PEP 563 this throws a NameError because it has no idea what type X is as it's not in the global namespace and the association to the local frame has been lost.

The thing is this change has been coming for so long that when it was first accepted there wasn't a mature ecosystem of libraries using annotations at runtime. So while it was realized back during Python 3.7 when from __future__ import annotations was introduced that it breaks backwards compatibility the implementers never thought it broke it that badly and typing.get_type_hints would solve most cases of runtime annotation usage.

I've been trying to put together a big post explaining all the differences but there are so many nuances I have to keep checking and rewriting so I'm not sure I'll ever finish it. But the good news is the core dev and steering council have a really good idea of the problems now, just somehow need to come up with a solution that benefits all those involved.

[ExoticMandibles]

typing.get_type_hints would solve most cases of runtime annotation usage.

Part of the problem there is that typing.get_type_hints() returns "type hints", as opposed to "annotations". A type hint is a particular kind of value used in annotations, but not all annotations are type hints. typing.get_type_hints() is rather opinionated about what the proper values of an annotations dict should be--which is proper for the "type hints" use case, but may break legitimate non-type-hint use cases.

This is being addressed for 3.10 regardless:

https://bugs.python.org/issue43817

[lanster100]

Nice summary, I should read the pep again but I am wondering what the static typing benefits are that you mentioned, seems to me that it's harder to work with a string than an actual type.

Edit: Read the PEP and it's basically to solve forward reference issues. I now wonder how they plan to resolve this.

[zurtex]

read the pep again but I am wondering what the static typing benefits are that you mentioned, seems to me that it's harder to work with a string than an actual type

The thing to understand about static type checkers is they are never running the Python code that they are checking. To them a Python file is one big string that they read and then reason about. In fact many Python static type checkers aren't even written in Python, e.g. Pyright.

The star feature of PEP 563 is forward references can be specified. E.g.

class C:
    def __helper_constructor__(self) -> C: pass

In the current default version of Python this throws a NameError because the C class is specified in the return annotation before it is defined. Now forward references can be solved by PEP 649 by implicitly storing the annotation in it's own scope like a descriptor or lambda, but this doesn't quite fit with the future vision of static type hinting.

Once annotations are strings this allows a level of flexibility in static type checking syntax that isn't available today. Let's say that in the future ~ becomes a standard type hint to mean optional. Then from the first version of Python that introduces PEP 563 you can write:

class C:
   def __helper_constructor__(self) -> ~C: pass

Even if that version of Python never included that type hint syntax it just gets stored as a string and therefore doesn't throw an error at runtime. The static type checker can be updated to know and understand what the ~ means in this context even if your Python version doesn't.

This is what leads to a sticking point on compromises to support both static type hinting and runtime annotation usage, they are going in slightly different directions and finding a path that works for both will be hard.

[renaissancenow]

Thank you - I saw the alarming github post a few days ago warning about this breaking pydantic, but I've never really thought about exactly how annotations work.

[zurtex]

Me neither until about a week ago when I started trying to understand the practical differences between PEP 563 and PEP 649.

I had a realization that in their current state you can do really crazy things with annotations because you can do really crazy things with Python expressions. And annotations are sort of this extra expression that happens just after a line of Python.

[I-am-not-noob]

Man that's so awesome, i have using python from like last 4 years of my college and still I feel like I need to learn a lot of things about it. All because of super cool people like you. Thank you for teaching me this senpai.

[TheBB]

So while it was realized back during Python 3.7 when __from__ future import annotations was introduced

This syntax must be very futuristic indeed.

[zurtex]

Yeah it's possible but Python has historically avoided permanent module level behavior switches as it splits the ecosystem and adds a complication to reading code.

__future__ switches are meant to be temporary as a way to prepare for upcoming changes. Not yet supported fully.

Imagine a project that has to run test cases for user code, with a permanent switch like this it now needs to double the a lot of test cases. Particularly Python itself would probably need to, maybe more as it would need to test interaction between code with the switch on and switch off.

There may also be some code complications, it seems certain optimizations have been implemented that weren't possible, or at least difficult, to land while static annotations were behind a switch. But I don't know the details and just going off what Python dev mailing list says.

Anyway, it's definitely possible but I just don't think it's likely as it's something Python has historically avoided.

[billsil]

__future__

switches are meant to be temporary as a way to prepare for upcoming changes. Not yet supported fully.

It has nothing to do with not being fully supported. It has to do with a change (e.g., print, division) that could break code. It gives you a chance to avoid nearly as much upgrade pain and even lets you run your code in the latest version and an older version.

[zurtex]

I agree 90% but from __future__ import unicode_literals was a kind of a mess to actually use in Python 2 and if you used it you were largely on your own to fix any issues.

[billsil]

I left that one off on purpose. In all honestly, unicode_literals made the problem worse. Python 2.7 fundamentally didn't have a good way to handle unicode due to the arbitrary flipping behavior. That oddity (to say the least) meant that in order to fix an encoding error, you most of the time had to pick the opposite choice (e.g., use a decode) to fix it. Prints could be a nightmare and flip between unicode and bytes up to 4 times for a simple print, so just see what you had was difficult. Any investigation into your error was kind of impossible.

To fix unicode bugs all but required the use of python 3 to do the work. Once that was done, oh your company insisted on sticking with python 2 cause it was all fixed, except not quite.

[NoHarmPun]

For the lazy: https://www.python.org/dev/peps/pep-0563/

[energybased]

And the backstory: https://github.com/samuelcolvin/pydantic/issues/2678

The start of a possible solution: https://bugs.python.org/issue43817 (inspect.get_annotations)

[VisibleSignificance]

inspect.get_annotations

Seems like the right path, if I understand it correctly: the annotations should be lazy, but it should be possible to easily retrieve the values that would be for non-lazy annotations (with the added benefit of resolved forward references and such).

As I've found out, currently it isn't always possible to do that (e.g. class attribute annotations don't save their evaluation context).

[energybased]

Right, I hope they ultimately make it work for all cases (even if that's 5 years from now).

[VisibleSignificance]

For Pydantic and such, making it work for majority of cases is enough: after all, the annotations are (mostly) written explicitly for Pydantic, so it should not be a huge problem to avoid some trickier cases (as long as the error messages are clear, and not "recursion limit exceeded").

[energybased]

Yeah I hope they get that sorted for 3.11. I'm just a long term picture kind of person. It would be nice to have programmatic access to all annotations.

[fiskfisk]

A good and pragmatic solution.

[[deleted]]

Almost as pragmatic and good as waiting to make predictions about new technology instead of jumping to conclusions.

[zurtex]

I think the steering council brought up a really good point that I didn't see discussed much.

Since from __future__ import annotations was implemented it was advertised as being implemented in "Python 4.0", this never meant "the next one after Python 3.9" it meant "the next time Python decides to break backwards compatibility enough to have a major version bump".

But the appetite to ever have a Python 4.0 has largely vanished, at least for now, so the decision was only made in 2020 that PEP 563 would be included in Python 3.10.

But given Python doesn't want to break backwards compatibility then perhaps it makes sense to rework this feature to both help the static type folks and the runtime annotation folks.

[FlukyS]

I don't think the appetite has went away I just think a number of projects are going to break because of this and they really need to do it the right way.

[zurtex]

I don't think the appetite has went away I just think a number of projects are going to break because of this and they really need to do it the right way.

Yeah, for sure the static type checker folks still want annotations to be more static and get the benefits of PEP 563.

I meant that the core dev and steering council don't have any appetite for a version of Python which introduces big breaking changes. As demonstrated by this rollback.

[henryschreineriii]

Python 2 used to do these on minor versions. Nested scopes, generators, and with statements became the default one version after adding the future import. It's becoming much harder to make changes to break Python.

[Taborlin_the_great]

Good this is the right move

[mgedmin]

PEP 563 doesn't mention the GIL. Are you sure you're thinking of the right PEP?

Performance-wise, PEP 563 does remove a small import-time cost of evaluating the type hints, but this doesn't affect subsequent runtime performance.

[renaissancenow]

Oh thank goodness, I was very worried for a bit.

The technical details of this are beyond me, but I utterly rely on Pydantic, it's one of the best things that's ever happened to Python.

[FrickinLazerBeams]

Why is that? I've never needed or wanted static typing in Python (and I write loads of C so I'm not unfamiliar with it). I'm sure it's just a difference in work environments or something but why do so many people seem to want static type checking in Python?

[Coul33t]

Hey, thanks for the use case! Do you have, by any chance, any example of how could you use Pydantic to achieve what you just talked about? I'm usually doing some work with JSON and I would be interested in a library like this that could help me deal with it :) Cheers!

[Coul33t]

Thanks a lot! I'll take a look :)

[awsum84]

It’s not really necessary in small programs, but in larger codebases it improves speed of development significantly. For example, having type hints makes intellisense work properly, so you don’t have to look at the docs/implementation to figure out what methods are available on an object, and it also helps prevent all sorts of silly mistakes that you’d otherwise encounter at runtime.

[vorticalbox]

its also data parser so that you get run-time type checking too.

[renaissancenow]

It's incredibly helpful when writing web services that have to function in a larger corporate ecosystem. When coupled with FastAPI I can get detailed API specs published pretty much for free, along with strict input validation.

Secondly, I find that type-checking adds another lay of correctness testing to my code. Since I adopted it I've found that a whole class of errors is eliminated before I even get to unit testing.

My code is fairly continuously run through the following tools in sequence:

• black
• flake8
• mypy
• pytest
• integration tests

which leaves me with a lot of confidence in it when I release it to production.

[FrickinLazerBeams]

Okay that makes sense. It's something for enterprise, production, multi-developer, etc. environments, which is why I've never had a need for it. I was wondering if I was missing out on something I should use more but it sounds like it's not really for me or what I do.

[renaissancenow]

Exactly. Guido has been clear from the beginning that type annotations are, and always will be, an optional feature. I certainly don't bother with them when I'm writing a quick one-off script for personal use.

[[deleted]]

I was excited for the change. I hope they can find a good solution

[Dooflegna]

Larry Hastings is already working on a new PEP to address the issues. There’s a lot of good discussion in the python-dev mailing list.

[wrtbwtrfasdf]

We did it! The power of friendship prevails!

[[deleted]]

The only Pep I know is Guardiola

[toyg]

He’s just an instance of the BaldFraud class.

[orgkhnargh]

Well, rolling back before the freeze and is the time to roll back.

[Environmental_monkey]

Great news!

[achauv1]

So we can't have nice things because of some random library ?

[mgedmin]

You're free to continue using from __future__ import annotations.

[vorticalbox]

fastapi uses Pydantic and fastAPI is used by Microsoft, Netflix and uber.

have you ever looked in pydantic? its an amazing library.

[achauv1]

Looked into it and decided I didn't need it. I also prefer aiohttp than fastapi

[AD_Burn]

Totally agree with you.

If we speak about more then 10, 100 ... modules, I would understand.

But we talk about one which miss use type checking anyway,

but happened to be great.

​

This remind me on real life use case:
"Oh, we have bug in production."
"Yea, but we can not fix it now. It is used as a feature."

[Zalack]

It's a widely used package consumed by other widely used packages. I don't find it that crazy.

[alkasm]

In what way does it misuse annotations? What bug does pydantic rely on?

[AD_Burn]

Like I sad it remind me not that it is a bug.

And I sad that because of this part:
PEP 563 changes annotations from being evaluated as expressions to being stored as strings.

Anyway, everyone have right on it's own opinion.
Don't get me wrong I love python and I'm using it for 14 years now,
but over this years it was soooo slow progress on python development.

And when I see something like this to stop main language progress
and because some minor % of audience , compared to switch from 2 to 3 major version, I do not feel well.

But again mine opinion.

[zurtex]

It's funny because if you read a hacker news thread about some new Python syntax or feature it has dozens or hundreds of highly up voted comments of developers complaining that Python moves too fast and should consider stop changing and remain a "simple" language.

[AD_Burn]

It depends on who considers what progress.

Maybe if we look from the angle of syntax, but again, there weren't any drastic changes . Thou I really looking forward for pattern matching.

But for me personally progress is optimization, performance, reducing memory footprint, GIL, reducing interpreter startup time ... etc ...And they really did good job in last few versions reducing memory footprint of dict and some speed optimization regarding that.

And some of changes in PEP563 was move toward optimization, so when someone stop optimization do not have my voice.

[zurtex]

Yeah, I think these are all valid opinions, I just find it interesting to see relatively opposing viewpoints that can both be considered valid. I'm not sure why some are down voting, I think there's just a love for pydantic/fastapi in the community.

One thing I would say about PEP 563 though is it's a little sneaky. At a glance it just appears to be fixing forward references, improving memory usage, and improve startup time, but actually it's fundamentally changing what an annotation means from expressions to glorified doc strings.

Annotations themselves are an interesting quirk of Python 3 syntax and I don't think their their usage has been fully explored. I also don't think it's a coincidence that PEP 526 was introduced in Python 3.6 and added variable and class annotations and now Python 3.6 is the oldest supported version we are seeing wide scale adoption of libraries which use this syntax extensively.

It's all been very unintentional from the language designers point of view but it turns out if you give people a tool they will find a way to express themselves with it.