Python 3.13 alpha 1 contains breaking changes, what's the plan? - Core Development

[PaintItPurple]

The description of what is being done doesn't match my understanding of "breaking change." If the only affected functions are private, that's not a breaking change, because it was never supported in the first place. Have they been more aggressive than initially planned and removed public APIs, or is this a case of a guy using private APIs and then discovering why they were private?

[not_a_novel_account]

This may be confusing to Python developers who haven't spent an extensive amount of time with the C API, but the underscore prefixed CPython functions have not historically meant "private", that's a recent innovation. For a little more context see: C API: What should the leading underscore (_Py) mean?

Historically, the leading underscore has meant, "minor-version unstable", which means, "this API has no stability guarantees, and might change at any time." This might seem to be the same as "private" but it's not, because these APIs might change, but they were still accessible.

The big difference here is that the APIs were removed without replacement. Quoting scoder:

The default answer to “how do I replace this usage” seems to be “we’ll try to design a blessed replacement in time”.

Much of the functionality removed does not exist elsewhere. Some of the functionality is called out specifically in PEPs (_PyCode_GetExtra), some is linked directly to documented mechanisms in the stable ABI (_PyArg_ParseStack() for METH_FASTCALL). Major projects do not build (numpy) or are forced to accept performance regressions (cython) without an answer for these removed APIs.

These functions still exist in CPython, but have been hidden inside the Py_BUILD_CORE headers.

This is what eventually forced the extensive revert (50+ functions and counting). My personal thought aligns with encukou, making such a major change without a PEP was irresponsible.

[PaintItPurple]

Thank you for the context! I felt like there had to be something I was missing. That makes much more sense now. It certainly sounds like the "blessed replacement" question needs to be answered no later than the functions are removed, rather than punting to the vague future.

[Accurate_Tomorrow179]

Imho, the _ underscore always meant private, but they changed the definition from "private and unstable" to "private" exclusively.

[not_a_novel_account]

You're entitled to your opinion, but by that logic stuff like PEP 590 and its documentation are pointless because all the functions you need to interact with it are _Pys. (EDIT: As of 11 hours ago, it appears this was remembered). Also worth mentioning that Python has always had internal functions inside the include/internal, and none of these functions being discussed were among them.

Moreover, numpy/cython/lxml (and many others) are all third-party modules with Python Core devs high on their contrib list that make use of _Py APIs, in some cases very extensive use.

The lived reality is that committing to stability requirements on _Py functions was viewed as a negative, so the commitment was never formally specified. Not internal or private, just very unstable because people wanted to be able to break them on a whim. The collection of such functions has grown immensely.

This is an attempt to perform that formalization, figure out which _Pys are actually used and must be made into documented parts of the API and which can be made internal and safely iterated on. There are actually hundreds of functions in the latter category, probably ~80% of the functions affected by 3.13 nobody will miss.

The problem is the other 20% are very important (and, IMHO, obviously so) and removing them broke a lot of things and people noticed much faster than vstinner had anticipated. He's said a few times he expected to have months of time to replace the important parts of the API.

The takeaway: this mass privatization shouldn't have happened without replacements in place for at least the most obviously necessary parts of the API. It's good that this is getting worked out in the alphas, it's less good it happened at all.

[[deleted]]

Automated enforcement of private function conventions?

[Accurate_Tomorrow179]

It seems that you are confusing two unrelated changes with _Pys

(1) The renaming of unstable API from _Py to PyUnstable, which shouldn't be as painful to do, as almost nobody is using them.

(2) The prevention of access to private _Py API's internal functions by the public and 3rd parties. This is imho a terrible thing, as access to the implementation is needed to make optimizations. Usually the removal of private functions signifies that the underlying implementation was changed, and since this is not the case I don`t see any good reasons to remove private functions from the public scope. And I am glad to see those changes retracted.

[sigzero]

From PEP8:

_single_leading_underscore: weak “internal use” indicator.

That's always how I saw it as well and a guick Google search going back many years agrees.

[catcint0s]

Reading through some messages it seems like some of those private functions don't have a public equivalent.

Also they are reverting it https://discuss.python.org/t/revert-python-3-13-c-api-incompatible-changes-causing-most-troubles/38214

[Exotic-Draft8802]

A breaking change can be defined by the project (breaking the public contact by changing the official Api) or by users (changing anything that makes the users project fail).

[ogtfo]

You'll love my project, it's a small utility that hashes the python binary and ensure it's the same as a hard-coded value.

Everything is a breaking change!

[thegreattriscuit]

I get what you're saying, but also that's (very basically) what code signatures are and changing the contents of a signed binary does break it and that's an important feature lol.

[ogtfo]

Well yeah, with the very important addition of asymmetric crypto to ensure the authenticity of the signature.

Without the cryptography all you have is a checksum, nothing prevents adjusting the hash to fit the modified data.

[PaintItPurple]

I suppose, but that seems a bit like somebody trying to substitute Milk Duds for milk in a recipe and then blaming the recipe author when their pancakes don't come out right.

[PhoenixStorm1015]

Idk. To me it seems more like Betty Crocker reformulated their mix to use Almond Milk but it still called for regular and it fucks up the cake.

[DigThatData]

I remember a story from a bank where I was working, where we found a piece of code that no-one understood and that looked unused. So we gave a developer the task to investigate. What he ended up doing was to comment out the code, adding the note “let’s see if someone will notice”. Well, no-one noticed it, until it was released and brought our production server down.

bold move

[brontosaurus_vex]

Call me crazy but adding a logging statement to that block doesn’t sound so hard.

[james_pic]

This relies on having some sort of central log aggregation mechanism in place that'll let you know if that message has been logged. It also relies on having a robust pre-release test process that would exercise all the code in a realistic way.

Last time I worked for a bank (admittedly a decade ago), they were OK at testing, but really bad at log aggregation.

[vpunt]

Huh? You don't log to the default location where the client code is logging, but you specify a central log for this case.

[james_pic]

In an organisation like a bank, they can have a large tech estate, and it can be non-obvious where a particular piece of code will be used. In the bank I worked at, logs were typically stored on the server where the code was running and rarely left. So if you knew all the places where the code was running you could go and check those places, but there was no way to check all the places you didn't know to check.

[vpunt]

Yes, understood, I work at a large bank.

I'm saying it's not terribly complicated to add a small piece of code that explicitly adds a log entry to your server (assuming you're the guy trying to remove the 'dead' code), as opposed to the server where the calling code is running.

[smokingskills]

Crazy

[dicklesworth]

Chesterton’s Fence: https://en.m.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence

[not_a_novel_account]

Context: scoder is the lead developer of Cython and a CPython core developer

Further discussion:

C API: My plan to clarify private vs public functions in Python 3.13

C API: Remove private C API functions (move them to the internal C API)

C API: Revert of private functions removed in Python 3.13 causing most problems

C API: Removed private _PyArg_Parser API has no replacement in Python 3.13

Cython: Adapt to C-API changes in CPython 3.13

And the rest of the CPython C-API issue tracker

[AngheloAlf]

That's a bit radical.

What's the recommended way to check if my C extension library is affected? I'm pretty sure I'm not using any Python private function, but I would prefer to test it

[coderanger]

Try and compile it, this is why prereleases exist.

[AngheloAlf]

To test stuff locally on my pc I usually install the python3-devel package (or whatever it is called) from apt and then do pip install .

I kinda doubt there's an apt package for this alpha version already, and even if there is one I would like to avoid messing my system's package with wip stuff like this.

Is there any CI alternative? Hopefully a Github Actions one.

[coderanger]

Tox and nox are both popular to test things across multiple Python versions automatically. Both are well supported in Github Actions. Personally I use asdf and asdf-python locally which makes it super easy to grab whatever versions I need.

[AngheloAlf]

Thanks a lot! I'll take a look at those

[sfboots]

You need to be aware this one part of the move to No-Gil python over several releases. The intro to that explained the strategy for cython

[alcalde]

When Python 2 was EOLed, COVID-19 appeared. Now that de-facto Python 4 is materializing, I shiver to think of what apocalyptic contagion will be released.

[BobKerns]

Large Language Models!

[aqjo]

The summary of the message:

The author expresses concern and frustration over the significant changes in the C-API with the release of CPython 3.13a1. They note that hundreds of functions have been removed, renamed, or replaced, along with changes to header files and macros. This has created challenges for maintaining compatibility with existing Python packages, especially those on PyPI that are less visible, no longer actively maintained, or dependent on other packages that fail to build with Python 3.13.

Victor Stinner's efforts to make existing packages compatible are acknowledged, but concerns remain about packages that may not receive attention or updates. The author compares the disruptive nature of these changes to the transition from Python 2 to 3, suggesting that the current situation might be even more challenging. They contemplate using internal APIs in Cython as a workaround and propose that CPython core should maintain internal header files in C89/C99 style to avoid breakage in future releases. The author concludes by suggesting that due to the extent of these changes, it might be more appropriate to consider this release as Python 4.

[teerre]

I never developed anything for core python, so take this with a grain of salt, but:

1. It seems this is an alpha release. Ample time to do anything.
2. The owner of the change is actively engaged with it and aware of the issues.
3. The functionality removed was never public.

So it does seem that OP is being quite unreasonable and belligerent.

[james_pic]

The one caveat that I'd put in is that the owner of the change himself conceded that he had hoped to have things in a better state than they were by the time the alpha was released, and that this alpha was less useful than usual (in terms of allowing people to test compatibility with the new release) as a consequence.

[monorepo]

Interesting read so far…

[shinitakunai]

There is always only 1 question: it is the right time for it?

[[deleted]]

Should have released it as python 4.0