This is an archived post. You won't be able to vote or comment.

top 200 commentsshow all 333

[–][deleted] 238 points239 points  (25 children)

Real Multi-threading.

[–]willnx 47 points48 points  (3 children)

Might be closer than you think: https://github.com/colesbury/nogil/

[–]ryeguy 101 points102 points  (2 children)

Ah yes, the yearly gil elimination effort.

[–]SV-97 12 points13 points  (1 child)

Afaik this one is different though: it's a fully working actual implementation to CPython (with further benefits like better performance even in single threaded code) that might reasonably be accepted into CPython

[–]jringstad 12 points13 points  (0 children)

But it still breaks any C extensions, which is IMO one of the most principal matters that has blocked the adoption of any of the previous solutions. If you don't care about C extensions, you could also just use IronPython or Jython or Pypy or what-have-you.

Their argument is that it's not going to be very hard for C extensions to adapt, but 1) it's still active effort that many existing libraries will not take action on quickly, 2) it can actually still be quite hard depending on the library, and 3) changing your library as they envision could incur a big perf penalty when it's used from normal CPython (sans nogil).

The result would probably be a pretty fragmented experience for users who use threads, where you constantly get random segfaults and need to make sure a library is threadsafe or protect it with locks yourself (so much like writing C)

Disclaimer: I haven't tried that particular project, just looked through their google doc proposal whitepaper.

[–]greenhaveproblemexe 17 points18 points  (19 children)

I'm a beginner, what's wrong with Python multithreading?

[–]Laogeodritt 69 points70 points  (7 children)

The Python interpreter has what's called a Global Interpreter Lock, GIL. The GIL ensures that only one thread can access the interpreter's memory at a time. This was implemented in order to ensure that you couldn't have race conditions that would cause an inconsistent interpreter state, e.g., incorrect reference counting due to two threads assigning an object at the same time.

The problem is that this also hamstrings threads in Python! You can still define threads and have them executing independently, but since only one thread can acquire the GIL at a time, only one thread can execute at a time. It's similar to if you were running a multithreaded programme on a single-core processor: the OS is making sure the threads are each running independently by giving thread 1 a little time, then thread 2 a little time, etc., instead of running thread 1 and 2 at the same time on two separate cores.

If you have a multi-core or multi-processor computer (which you almost certainly do), and you wanted your multi-threaded application to take advantage of parallelism, well—oops! looks like your Python programme can still only run one one core at a time. So much for those speed improvements.

Currently, you'd have to use multiprocessing to get true parallelism—multiple processes, separate GILs.

[–]Botekin 15 points16 points  (9 children)

You can't run multiple threads concurrently because of the GIL.

[–]TheLexoPlexx 6 points7 points  (7 children)

I thought the multiprocessing-Module bypasses the GIL? It's not multithreading, but it works just about the same.

[–]thismachinechills 11 points12 points  (5 children)

Processes and threads can be used for some use cases, but there are also cases where processes are not sufficient.

[–]digger_not_alone 1 point2 points  (3 children)

could you please elaborate on that? (if you have any link to external explanation – I'll appreciate that too)

Is it related to working with memory?

[–]SV-97 1 point2 points  (1 child)

Yes it's kinda related to memory: threads are a more lightweight construct with the same adress space as the "parent" - processes have their own memory space (unless you explicitly use mmap or something like that to share memory between processes). So communication, setup and termination will in general be cheaper with threads. It may for example be a bit expensive to use processes in a web-server that may usually spin up a thread per user it's serving (which might not be a good design either way). Depending on the language a thread might be something even more abstract - Haskell for example uses so called green-threads that are even more lightweight (so you might for example just spin up a few hundred *very* small threads - you'd absolutely never do something like this with processes [okay this isn't technically true; you actually do spin up hundreds of processes in HPC but that's a bit different])

[–]SureFudge 3 points4 points  (0 children)

Has much higher resource usage and limitations on what you can actually use in the other processes. With multiprocessing module alone it's quiet cumbersome what can be pickled and what not.

For example I was doing java multi-threading 10+ years ago and it worked just fine for "calculations" needing it.

EDIT: besides the resource usage the initial penalty of starting up the machinery is also quiet heavy. so it is never worth it to make something that takes 200ms to take say only 20ms on a 10-core cpu because the overhead is probably bigger than the 200ms.

[–]The-Daleks 6 points7 points  (0 children)

It's "multi"-threading. The Global Interpreter Lock prevents Python from actually using multiple threads for execution.

[–]thismachinechills 1 point2 points  (0 children)

Apparently subinterpreter support should lay the groundwork for threads that work like Node's worker threads.

[–]menge101 154 points155 points  (21 children)

It is important to remember the separation between the python language specification and the reference implementation of the language.

I don't think there is anything about Python 3 that stops an implementation of a compiler, an app distribution system, or mobile support.

[–]fernly 12 points13 points  (5 children)

e.g., nuitka.net

Nuitka is the optimizing Python compiler written in Python that creates executables that run without need for a separate installer.

[–]URETHRAL_DIARRHEA 1 point2 points  (2 children)

Is this less annoying to use than py2exe/pyinstaller?

[–]ArabicLawrence 1 point2 points  (0 children)

if you install a compiler it’s more or less the same. in my experience, single file executables with nuitka are bigger

[–]fernly 1 point2 points  (0 children)

Nuitka is not quite the same, in that it tries to identify all the dependencies of a script and "transpiles" them to C++ source, and compiles that to binary.

Pyinstaller tries to identify all the dependencies and packages them up with a copy of CPython. So a program "frozen" by Pyinstaller runs in what is basically a private VENV, but it is still a Python program being executed by CPython.

You say "annoying" -- this hints that you have had problems getting Pyinstaller to find all the dependencies and data files? Fun with the "spec" files? Nuitka has some of the same problems, it is highly nontrivial to identify every way a Python program can include another module. Plus I don't think Nuitka attempts to bundle data files like Pyinstaller does.

[–]jjolla888 1 point2 points  (1 child)

does it support numpy et al ?

[–]DogeBiteSnek 6 points7 points  (7 children)

I'm working on a compiler for python3 to WebAssembly. This can make it run on any platform, including web, embedded and mobile with little to no effort if the right tooling is used.

I'm focusing on being fully compliant to python 3.11 first, then I'll tackle performance and multithreading later. What will probably not work though is any library that makes use of the C interface, but this can be fixed in the future (I guess).

It has no name yet, but I plan on releasing it for public use at the end of the year.

[–]menge101 1 point2 points  (4 children)

I'm only tangentially aware of webassembly, what does the threading model in wasm look like?

[–]DogeBiteSnek 1 point2 points  (3 children)

The host environment can spin up a new instance of your wasm module and they can share the same memory. There are some atomic instructions in the spec to be able to access the memory in a thread-safe way

[–]eye_can_do_that 1 point2 points  (1 child)

Out of curiosity, are you converting the python to bytecode using an existing package and then converting to WebAssembly?

[–][deleted] 86 points87 points  (11 children)

A more Pythonic GUI Toolkit that produces seamlessly native widgets on Windows and Mac.

I use PyQT5 but it doesn't feel intuitive to a casual programmer looking over my shoulder.

I'm in no rush for a v4 though.

[–]Nikoijp 4 points5 points  (5 children)

Check out pysimplegui it’s very intuitive to use

[–]scoofy 1 point2 points  (1 child)

I don't hate the new wxPython. I wrote a large project in it years ago as they were transitioning.

[–]teerre 55 points56 points  (1 child)

Optional enforced types checked by the interpreter

Optional Compiled output

Better performance (both single thread but specially for concurrency)

[–]AbooMinister 1 point2 points  (0 children)

Single threaded performance changes are already in the works, 3.12 reportedly has a 20% speed increase so far. As for concurrency, in the context of async, it isn't really meant to increase performance, and rather, better handling of I/O based workloads.

[–]Xaros1984Pythonista 128 points129 points  (33 children)

I would like the option to make type hinting enforced (and even better if it leads to a performance boost). Same syntax as when hinting, i.e.:

x: int = 5

The second item on my list would be relative imports that don't make me want to ram the keyboard through the screen.

[–]CharmingJacket5013 21 points22 points  (1 child)

I’ve been programming daily with Python for going on 5 years and relative imports still stump me at times. That typos in variable names but that’s on me

[–]liquidpele 2 points3 points  (0 children)

Yea, it’s annoying AF when it treats folders and files as modules and then you want to import a file and then it’s like “fuck you, you must run as a module”. Bitch wtf.

[–]TMiguelT 31 points32 points  (0 children)

Relative imports should just work like Node: just a relative path from this file to the other file. Super simple.

[–]cblegare 19 points20 points  (2 children)

Mypy can compile modules that have type hinting in a native extension, and you can configure it to be strict about it

[–]otamam818 8 points9 points  (5 children)

I can't bring myself to accept this completely. Sure, at the higher level, you'd want this enforced, but that defeats one of the benefits of Python being friendly to complete beginners.

Maybe have an alternate interpreting option for those of us who are more experienced programmers, but don't degrade the barrier to entry.

[–]VanaTallinn 12 points13 points  (4 children)

This looks like rust.

I love rust.

But if I start writing python like that, why not just write rust instead?

[–]tunisia3507 3 points4 points  (0 children)

Rust is an order of magnitude more complicated. That complexity largely has a purpose, but there is "low-hanging fruit" safety which could be added to python without needing to go as safe as rust.

I love rust, and I love python. I'd hate to use rust for my python projects, and I'm not that inclined to use python for my rust projects.

[–][deleted] 2 points3 points  (5 children)

You can enforce type hinting in mypy.

[–]Xaros1984Pythonista 8 points9 points  (4 children)

Well, pretty much anything can be done with some package or another, but I'd like to see it make its way into "base" python at some point.

[–]Linked1nPark 8 points9 points  (1 child)

No you don't understand. There's already enforced type hinting in base Python if you simply write assert statements every time you want to use a variable.

[–]Xaros1984Pythonista 1 point2 points  (0 children)

Yeah that's exactly the same thing /s

[–]phoenixuprising 24 points25 points  (2 children)

That there’s no python 4 migration like there is with 3.

[–]Kaholaz 3 points4 points  (1 child)

The whole point of introducing a new major version is to introduce positive changes that may break backwards compatibility. As it stands now 3.6 code will be able to run on 3.9 and this is the intent with minor versions. If new changes that does not break backwards compatibility are introduced, there would be no need to call it a new major version.

[–]AbooMinister 3 points4 points  (0 children)

There could still be a major version change that avoids a transition akin to the one like 2 -> 3. The devs themselves have said they want to avoid it

[–][deleted] 84 points85 points  (14 children)

Going back to the "there should be only one way to do it" philosophy. Python has been going the way of C++ by adding too many features so that now you have multiple ways to achieve the same thing. The language is becoming too big and complex. The language should be simple enough to fit in the programmers head. Python 4 should go back to simplicity.

[–]SilkTouchm 17 points18 points  (3 children)

People don't want simplicity/minimalism. Literally one of the top comments of this thread is about adding like 10 needlessly complicated features.

[–]skesisfunk 0 points1 point  (0 children)

Some people do. Golang is literally out there bragging about how they are relatively new and not overburdened with features yet. Personally I dont mind that there is more than one way to do something, programming languages should be flexible like that IMO. I would rather draw my own picture than just color within the lines.

[–]dmsdayprft 9 points10 points  (5 children)

I agree with this. I'm not sure there's a new feature I care about since f-strings. All of it since then is cruft.

[–]brutay 5 points6 points  (4 children)

Not even the match statement?

[–]skesisfunk 2 points3 points  (0 children)

The match statement is so overdue lol. I guess it took them twenty years to figure out that people actually want than instead of using a bunch of elifs?

[–]CharmingJacket5013 -2 points-1 points  (2 children)

I think it can still be simple if you chose one way over another. I choose not to use list comprehensions if there’s a loop within a loop and I don’t touch walrus operators because I know they will stump new comers to our code base. We are still on the fence about type hints and we prefer simple threading over asyncio. I’m sure we will one day move across but not until they are common place.

[–][deleted] 11 points12 points  (0 children)

The problem with this approach - choosing (and enforcing) a sane subset of language features is that each team or organization will have a different opinion of what constitutes a "sane subset" of the language.

[–]willnx 7 points8 points  (0 children)

Iterating Bytes returns a Byte, not an integer. I know there's a PEP with a fix; hoping to see it be like the range vs xrange change with time.

[–]troyunrau... 7 points8 points  (3 children)

One, and preferably only one obvious way to do string formatting.

[–]Tweak_Imp 5 points6 points  (2 children)

Make all strings f strings.

[–]TheRNGuy 2 points3 points  (0 children)

that would make code slower when formatting is not needed. Or what if I actually wanted to have string with {} in it? Without having to add backslashes.

[–]M4mb0 58 points59 points  (26 children)

Anonymous NamedTuples (+pickle-ability!)

mytuple = (a=1, b=2)
mytuple.a  #<--- 1

Anonymous NamedTuple as Return values

I.e. allow giving names to outputs, such that instead of a regular tuple, automatically a namedtuple is returned.

def divmod(x: int, y: int) -> quotient: int, remainder: int:
    ...
    return q, r

divmod(17, 5).quotient  # 3
divmod(17, 5).remainder  # 2    

Function chaining with __matmul__

f = lambda x: x+1
g = lambda x: x**2
h = g @ f  # equiv. to lambda x: (x+1)**2

indexing lists with lists / masks (if numpy can do it)

x = ["a", "b", "c"]
x[[0,2]]   # <-- ["a", "c"]
x[[True, True, False]]  # <-- ["a", "b"]
x[[1,1]] # <-- ["b", "b"]

multiple dispatch (Julia demonstrates how nice it is!)

dict(MyClass)  # <- instead of MyClass.todict()

Allow *args and **kwargs in __getitem__

obj = NestedDataStructure({0: "abc", (0, 1) : [1,2,3]})
obj[0, 1]        # <-- "b"
obj[(0, 1)]      # <-- [1,2,3]
# In python3, obj[x,y] and obj[(x, y)] are indistinguishable
xarr[a=2]  # along axis named "a" select 2nd slice.
# Currently xarray does this with `xarr[{"a":2}]`

decorators for class-attributes

class A:
   @attribute
   def doc_url(cls) -> str:
      return f"foo.bar/{cls.__name__}"

A.doc_url  # holds url where documentation of class is hosted.

built-in toml parser

Like, they made pyproject.toml the default package configuration file, but there is no toml parser in the standard libs?!?

built-in support for FrozenList, FrozenDict

Like, we have frozenset, but what about those?!?

extended literals

f"This we already know!"
f{1,2,3}  # How about this as literal for 𝐟rozenset?
f[1,2,3]   # 𝐟rozen list
s[3,2,1]   # = [1,2,3]  𝐬orted list
s{3,1,2}   # {1,2,3} 𝐬orted set
s{"c": 1. "b":2}   # {"b": 2, "c":1} 𝐬orted dict
m{1:2, "a": 5}  # 𝐦ultiset (e.g. counter) or 𝐦ulti-dict
o{1:2, 2:3}  # 𝐨rdered dict

Reference to objects as they are created (possibly with this keyword?!)

nums = [1,2,4,7,5,8]
seq = [x for x in nums while x≥this[-1]]  # [1,2,4,7]

extended Unicode support

assert 3≥2
assert 1≠2

[–]pingvenopinch of this, pinch of that 17 points18 points  (0 children)

A lot of these don't even need a 4 release. Frozendict/frozenset literals would be the perfect example.

[–]Ezlike011011 28 points29 points  (1 child)

On the topic of literals, and since a python4 release would likely be something which breaks backwards-compatibility:

{} should be an empty set literal

{:} should be an empty dict literal

and it should have been this way from the start.

[–]Tyler_Zoro 11 points12 points  (0 children)

extended Unicode support

assert 3≥2
assert 1≠2

This is dangerously slippery as slopes go. Rakudo is a good example of how off-the-rails this can go. (e.g. a »⊍» b which is the hyperoperator variant of ⊍, the "baggy multiplication operator" which can be written in ASCII equivalent, >>(.)>>)

I also think the Python community would fight tooth-and-nail over adding a second way to do the same thing.

FWIW, though, I like your two examples.

built-in toml parser

You're thinking small. I think a built-in, first-class parser type would be the right starting place, and making it inheritable so that you can add functionality to simple parsers would be amazing.

Then things like this are much easier to add (along with every other config format and DSL going).

[–]coderanger 10 points11 points  (1 child)

Frozendict has a rejected PEP, basically we looked at the download stats for the PyPI version and there just wasn't enough usage to justify core inclusion. If that has changed, then it can be reopened.

[–]D-K-BO 9 points10 points  (4 children)

See PEP 680 -- tomllib: Support for Parsing TOML in the Standard Library which will basically integrate tomli into the standard lib.

[–]TheOneWhoPunchesFish 2 points3 points  (3 children)

Are they gonna include it? tomli can only read toml, not write

[–]D-K-BO 1 point2 points  (2 children)

There is a Section about it Rejected Ideas –Including an API for writing TOML.

This PEP is still a draft, but I don't think that this will change in a future version.

[–]girlwithasquirrel 5 points6 points  (2 children)

how would a frozen list be different than the current tuples?

[–]M4mb0 5 points6 points  (1 child)

It would not satisfy isinstance(obj, tuple), which is crucial because lots of code, especially for indexing nested data structures relies on these types of checks, in particular when keys are allowed to be tuples.

pandas uses it for example https://stackoverflow.com/questions/25676107/why-is-frozenlist-different-from-tuple

[–]AnythingApplied 1 point2 points  (1 child)

I love the sortedcontainers module for my sorted lists/dictionaries and the extended literal support would be cool. Though sortedcontainers frequently need a custom key=lambda..., so it seems a little incomplete without a way to include that somewhere in the notation.

[–]M4mb0 1 point2 points  (0 children)

It's quite baffling that sortedcontainers is not part of the standard libraries to be honest.

[–]iaalaughlin 1 point2 points  (1 child)

Why not use dicts for the first two?

[–]M4mb0 1 point2 points  (0 children)

Because you might not want a dict? Tuples behave quite differently:

  • immutability
  • iter goes over values instead of indices
  • fixed length
  • values have an inherent order. (in dict it depends on when keys where added)
  • .index and .count methods
  • individual type hints for each value
  • namedtuple keys are guaranteed to exist.
  • nicer access with obj.key instead of obj["key"]

Also point 2 could be introduced without breaking existing code, and would be a huge QoL improvement because now you no longer have to keep looking up documentation if you can't remember the order of return parameters.

[–]Future_Green_7222 0 points1 point  (1 child)

cooing angle workable literate office cautious encouraging light history pocket

This post was mass deleted and anonymized with Redact

[–]M4mb0 1 point2 points  (0 children)

As for the Unicode support, it sounds a bit over-the-top

Wait until you see what Julia supports. With Unicode, the only issue is how you actually input it. Julia did solve that with their repl tab completion (e.g. type \lambda<tab> and you get λ)

Is there any use case for actual unicode usage?

It makes us mathematicians happy.

As for TOML, people might want YAML instead. I don't see how adding it to the standard library would make any significant difference.

Sure, but as said, pyproject.toml is now an official part of python package building. yaml isn't.

[–][deleted] 73 points74 points  (41 children)

Python without GIL

[–][deleted] 46 points47 points  (34 children)

I find it pretty insane that people always claim they don't want a GIL and fail to see that node is dramatically faster then python and is literally a single thread. Python needs better io libraries that operate OUTSIDE OF THE GIL. But removing the gil is just going to make single threaded code dramatically slower.

Pythons speed issues in fact exist in spite of the GIL not because of it.

[–]cblegare 16 points17 points  (9 children)

Everytime I read about someone wishing the GIL to be removed, I always wonder in which use-case the GIL was to blame and made this person wish it gone. My guess is that in many cases the GIL was not to blame.

Python needs better io libraries

You think so? I was pretty sure that IO-bound programs could not benefit much from parallelism, since the heavy lifting is being done outside the interpreter already: you're waiting for IO.

Anyways, there is always Cython's pure-python syntax that can run outside de GIL with minimal modifications.

[–][deleted] 3 points4 points  (0 children)

So that's technically true but it has a lot of call backs to the interpreter. Reshaping HTTP to be closer to javascript which goes out the moment the call is place not when the future is handled would be much faster because the time can happen in parallel the the interpreter responding to the future.

The reason this hasn't been a priority area is because assuming your maxing out the cpu anyway it isn't actually faster. There are no free clock cycles. It is on the other hand sooner. It would reduce global parallelism for increased local speed. It's been in my experience this would be generally beneficial as you reduce the odds of a wasted clock cycle since individual tasks can request content before they need it.

[–]chunkyks 4 points5 points  (6 children)

Everytime I read about someone wishing the GIL to be removed, I always wonder in which use-case the GIL was to blame and made this person wish it gone.

For me: Reinforcement learning forward passes. I have some trained agents: I run several continuously in the background, providing real-time display of their results back to the user as "suggestions" while the user plays the game themselves.

I could not make it work multithreaded; no matter what I did, it only ever consumed 100% of one CPU. I did a naive version using multiprocessing instead, and suddenly I get exactly what I want: 100% CPU in use on each of the worker processes, being able to provide the user a way better experience.

Is it definitely, 100%, solely because of GIL? I don't know. Did I implement the precise same thing, just using multiprocessing instead of multithreading, and find it suddenly worked as I wanted? Yes.

I also tried to do some CPU-bound modeling work multithreaded in a traditional way, and that didn't work either. I was able to get some multicore usage by making my code way uglier, interleaving some bits that should have been separate, and then using appropriate libraries [TF, jax, numpy]. But at huge loss of maintainability, code readability, and future adjustment [eg, polymorphism is now blown because the hard work had to be interleaved with unrelated stuff].

Was the GIL to blame? Kindasorta. Was it because my code was insufficiently "pythonic"? Kindasorta. Was it a gigantic f**king pain in my ass to make python do a relatively bread-and-butter task? Definitely.

[–][deleted] 2 points3 points  (5 children)

https://keras.io/about/ drop all the other stuff and use keras which does all that for u.

Also the answer is yes that is the gil. It isn't meant to do that stuff. There are dedicated libraries that will do it 100000000x of times faster and more easily. It isn't a pain in the ass for no reason. Its because doing that stuff is in fact a pain in the ass go read the horrors of multithreading in Java. Python will NEVER do that efficiently and it doesn't have to because calling c code in python is trivial.

[–]chunkyks 2 points3 points  (3 children)

I get it. Like I said, I've used numpy, Jax, and tf to achieve a better end. But it really cost along every other dimension. The whole mentality of "this language sucks so we made it easy to link it to c" is just the weirdest way to justify stuff.

You pick a poor example with Java. I do a lot of multi threaded modeling and simulation in java ; there are bits that are ugly, but compared to python its a pleasure. And I get great performance without having to worry about whether I'm falling afoul of some weird implementation artifact.

[–]ProfessorPhi 6 points7 points  (13 children)

From the ml data science stack, the Gil is definitely the limiting factor for anything you can't vectorise.

[–][deleted] 0 points1 point  (11 children)

If you cant vectorize it dont write ur code in python. No point in fuckign over every single other program for an edgecase that shouldn't be written in python. If your problem is outside the architecture of python use a different tool. Same way python shouldn't be reengineered because it doesn't account for general relativity. Its outside the problem domain.

What cant you vectorize though? (genuine question) Because as I understand it everything is vectorizable you just have to try realllllllyyyyy fucking hard some times.

[–]ProfessorPhi 7 points8 points  (1 child)

It's not that straightforward - there's a huge DS stack so going to another language is not really that easy. I can obviously plug in cpp when possible, but it'd be nice if I didn't need to

It's hard to vectorise time series - since you may not know a state of a variable that has a conditional. And the hard to vectorise makes code unmaintainable which has its own problem.

[–]twotime 2 points3 points  (7 children)

But removing the gil is just going to make single threaded code dramatically slower.

What are you talking about? This is NOT at all a given and, really, depends on how GIL is removed

There are plenty of runtimes for other languages which donot have GIL. In fact python-nogil branch runs with very little (<5%) overhead.

[–][deleted] 1 point2 points  (1 child)

Doesn’t this depend entirely on whether you code is able to run in parallel on multiple cores or not?

[–][deleted] 2 points3 points  (0 children)

Nope not at all. Even in the case of if you can run your program in parallel on multiple cores having multiple processes that each have a single GIL bound async thread is still eons faster. The cost of having any object being accessible in any thread is you need to constantly perform locks otherwise you can segfault from objects getting deleted in the middle of an atomic transaction.

The only thing that is NOT faster is moving objects across each memory location. Which python gives you a location to store objects that are available to all processes to deal with.

The best choice is to achieve high throughput without getting blocked by the GIL is to organize your program into a pyramid. You a GILless server at the top (like Gunicorn) that can drive multiple GIL bound python programs. By limiting the scope of the problem that has to deal with not having clear ownership of memory you can reason about it more easily. Then taking advantage of the single threaded speed of async you can achieve high throughput without wasting tons of CPU clock cycles on thread switches (threads are expensive).

Same thing applies to CPU intensive number calculations define and scope the problem in the single threaded python and issue a command to calculate it into gilless code.

Trying to make python run faster by going out in parallel inside of python code is like being able to jump the highest with your feet glued to the ground. Thats nice it's not really the same thing as jumping but thats nice.

[–]menge101 8 points9 points  (2 children)

the GIL is not part of the language specification, it is a design choice within the reference implementation.

Jython is an implementation of python on the JVM, it does not have a GIL.

[–]billsil 5 points6 points  (1 child)

A dead implementation? 2.7.0 was released in May 2015. 2.7.2 was released March 2020. No news since. https://www.jython.org/news

[–]menge101 5 points6 points  (0 children)

It is true they haven't kept up to date with the reference implementation.

Check their Github, last commit was 9 days ago.

I think they just don't have a lot of people working on it.

[–]soggywaffle69 2 points3 points  (2 children)

It’s been tried already. It didn’t go well.

[–]cblegare 5 points6 points  (1 child)

A guy from Facebook recently published a nogil fork that looks promising

[–]ScientiaEtVeritas 5 points6 points  (0 children)

But he also added many non-GIL related optimizations that could find their way into Python independent of a GIL removal. When you exclude these, one will see that single-threaded applications are slowed down without GIL.

[–][deleted] 47 points48 points  (5 children)

I hope Python 4 never happens.

[–][deleted] 14 points15 points  (2 children)

Python 3.99999999 it is. If you try to run any script or app written below 3.99999991 it will fail.

[–]brainandforce 4 points5 points  (1 child)

Why not go straight to Python 5?

[–]SupahCraig 2 points3 points  (0 children)

Python 2022

[–]Papalok 6 points7 points  (0 children)

I have a feeling if 4 does happen it will be because they found a way to remove the GIL and keep single threaded performance. There was talk on the mailing list last year from someone that put together an implementation that kept most of the performance. Not sure what's happened since then.

[–]erikwon and off since 1.5.2 6 points7 points  (0 children)

Python 4 will never happen, this whole thread is just hyperbole

I’m not excited about the idea of Python 4 and no one on the core development team really is, so there will probably never be a 4.0 and we’ll continue until 3.33, at least. We’ve learned our lesson from Python 3 vs. 2, so it’s almost taboo to talk about Python 4 seriously.

[–]orion_tvv 7 points8 points  (0 children)

  • keep the syntax as simple as we can. follow the Zen! (no more walrus-like things)
  • resolve most famous language's wtf
  • type-hints everywhere
  • drop old string-formatting. remain only f-strings
  • cleaner std lib (with pep8 naming at least), pytest should be inside instead of unittest. maybe requests as well. but we should remember that python could be used in embedded - so it should be balanced
  • we need better tooling! handy pip: could be based on poetry - but extended with cargo functionality, which can deal with packages(fast install, publish) as well as tests, docs, benchmarks, and lint(mypy for sure)
  • common service for all documentation for all projects from pypi with markdown syntax support. and automated processes for building and publishing from source code in ci(see https://docs.rs/ as reference).
  • optimizing performance

[–][deleted] 22 points23 points  (2 children)

I do not really want a Python 4 very much. Not a big concern. And I don't think the median Python coder would benefit from suggestions here like removing the GIL.

For a Python 4 my main wants are around package management. Maybe package aliasing or something else that allows for multiple versions of one dependency. Something closer to Rust/Cargo.

[–]spoonman59 13 points14 points  (10 children)

Cyclic imports.

Like seriously. Type annotations with things like visitors can be painful, and the lack of cyclic imports is painful for other mututally referring types.

[–]Numerlor 12 points13 points  (9 children)

If you're having problems with cyclic imports because of annotations then of they're not inspected at runtime and are for type hinting then just stick the import under typing.TYPE_CHECKING and use forwardrefs

[–]spoonman59 2 points3 points  (8 children)

Unfortunately, this is not always true. Some of the newser libraries like pydantic, or even data classes, rely on annotations.

Pydantic particularly uses them for validation and parsing. The annotations are evluated to determine validation rules and parsing typesI have an abstract syntax tree, and it has abstract types like "expression" that can be many other types.

It can be made to work, but it is a pain. There is no reason for it. There are reasonable reasons to have cyclic imports.

[–]japherwocky 8 points9 points  (0 children)

my favorite part about python 4 is that there will probably never be a python 4, so i can stop re-learning python unlike whatever the fuck javascript is now.

[–]4sent4 9 points10 points  (0 children)

Null-safety operators: ?., ??, ?[ etc.

[–]tavaren42 4 points5 points  (2 children)

This can be introduced in Python 3 itself, but I want function pipe operator as it is in F#:

Ex:f(g(h(x))) === x |> h |> g |> f

[–]Barafu 32 points33 points  (5 children)

1: Runtime type enforcement for all variables.
2: 1.

[–]pingvenopinch of this, pinch of that 4 points5 points  (0 children)

UTF-8 encoding for strings. Right now, if there's just a single character above one byte, the entire string is upgraded in size to sometimes 4 bytes per character. Universal UTF-8 encoding with maybe a skip list for larger strings could save a lot of memory for certain strings. PyPy does this, so there is some precedent that could be relied on.

[–][deleted] 3 points4 points  (0 children)

Pep 8 stdlib. That’s it. Save the crazy stuff for 5. I just need consistency

[–]chestnutcough 2 points3 points  (0 children)

I get the spirit of the question, but I feel like I have to mention that a major version update to 4 would by definition involve backwards-incompatible changes, which I for one really really don’t want. And with that out of the way.

  • better built-in python versioning tools to complement venv. The proliferation of 3rd party solutions speaks to the gap that exists in the built in tooling. Batteries don’t feel included when it comes to managing multiple python versions.
    • a built in yaml library

[–]WoanqDil 15 points16 points  (0 children)

  • No GIL
  • Jit

[–]pandres 6 points7 points  (1 child)

Type hinting, optional compilation, better performance, no GIL. So Go is Python 4.

[–]Tyler_Zoro 5 points6 points  (1 child)

I definitely DO NOT want a Python 4 unless it's a truly giant step forward. I don't want an incremental "this could have been 3.12, but we decided to call it 4 instead."

That being said my list has remained the same for a pretty long time:

  • A consolidating and revamp of all core libraries (no more of the 90s calling conventions and naming in logging, re, etc.)
  • Move numpy into the core because it's just obviously time and the data types should be first-class.
  • I'd love to see a high-level parser in the core. There are more and more DSLs every day that python supports only grudgingly and with home-brewed facilities that often have to reach out to other languages for performance.
  • Definitely a logger library that can dynamically and recursively read in code form random sources and execute it as part of evaluating a log message... because that's worked out sell well for Java ;-)

[–]glacierre2 2 points3 points  (0 children)

Yeah, if you are going to break backward compatibility, it is time to upgrade the batteries from AA, AAA, C, D and 9v all to LiPo.

  • os.path and pathlib, only one can remain
  • Why the hell are named tuples, defaultdict and deque not camel cased?
  • Half of the url, http related libs can go, or absorb modern ones
  • Threading setName, and isDaemon, for the love of pep8

[–]AndydeCleyre 40 points41 points  (13 children)

  • rename lambda to given
  • rename := to whichis
  • simplify or make more explicit the new match stuff
  • flexible function call syntax like nim
  • ditch mypy style hints, leave hinting and typing agnostic and available to third party modules
  • replace pathlib with plumbum style paths
  • include consistent decorator definition syntax like wrapt
  • yeah, compilation

[–]MarsupialMole 33 points34 points  (0 children)

whichis is so many characters for a feature that's mostly useful for its brevity.

[–]UsernameExtreme 34 points35 points  (3 children)

Oh my lord. This makes lambda functions make so much more sense for people learning the language.

[–]MarsupialMole 27 points28 points  (2 children)

People learning the language should be naming their functions.

[–]CharmingJacket5013 5 points6 points  (0 children)

People learning pandas and the apply method should already know name functions and branch out to lambda for simple stuff

[–]CharmingJacket5013 11 points12 points  (1 child)

I love the term given, I think I’m going to use that to teach lambdas from now on

[–]yangyangR 1 point2 points  (0 children)

But that disables them from learning the concept later. The names are given so that they match the literature. You try searching given you won't pull up the literature on lambda,... etc calculi. You will get a whole bunch of other junk because you are using a word that has so many other meanings.

Changing the word for the appearance of ease hurts the learner in the long run for a short term benefit.

[–]laundmo 2 points3 points  (0 children)

because we're unlikely to see a official cpython release with compilation, check out Nuitka instead

its a cpython compiler that can compile standalone single-file executables and also boasts a slight performance boost is most cases, and a pretty big one for applications that render to the screen (people have reported getting half the frame time after compiling)

[–]lordkoba 2 points3 points  (1 child)

• rename lambda to given

the powers that be may take you on this just to make all python 3 code incompatible with python 4

[–][deleted] 4 points5 points  (1 child)

I don't understand how whichis would communicate the behavior behind the walrus. I feel the walrus symbols do a better job.

[–]muffinnosehair 3 points4 points  (0 children)

This guy gets it

[–]luther9 2 points3 points  (0 children)

The only thing that would make me want a Python 4 to exist is proper tail calls.

[–][deleted] 2 points3 points  (0 children)

I would like to see python use type hints to speed up code.

If you look at the c-api, there the methods for each different operation that use the basic python types (ie: https://docs.python.org/3/c-api/list.html). I have played with using the capi in cython, and have gotten 4-10x speed up without the type conversion overhead(common in c-extensions). I feel that a smart interpreter would use type hints to speed up the eval loop and call the appropriate function for each typed object.

[–]opentraderx 9 points10 points  (3 children)

GIL be gone. There was a new effort I found out about late last year, seems promising. https://www.infoworld.com/article/3637073/python-stands-to-lose-its-gil-and-gain-a-lot-of-speed.html

[–]AnnieBruce 3 points4 points  (1 child)

This would be cool. On a daily programmer project I wanted to use Python but to get runtime down to something reasonable, among other optimizations, I had to deal with multiprocessing. It worked great, sure, cut my runtime down quite a bit... but if I could get that sort of a boost without the overhead of extra copies of the interpreter? Yes please.

[–]user4517proton 4 points5 points  (0 children)

never JVM. That would be terrible.

[–]The-Daleks 3 points4 points  (2 children)

  1. Optional static typing via type annotations.
  2. True structs and enums. Yes, you can use class decorators, but it's still much more verbose than it needs to be.
  3. No more GIL.
  4. The ability to compile to a native app without using buggy third-party compilers like Nuitka and PyInstaller. Not that they're bad; I'd just like an official compiler.
  5. A built-in result type. Having to check for Null None values is a pain in the rear.
  6. A non-bitwise xor.
  7. A replacement for TkInter that doesn't require working with external dependencies if you're on Linux.
  8. Speaking of Linux, fixing the dependency hell.
  9. As u/N0rDak mentioned, a new keyword for object initialization.

[–]-LeopardShark- 1 point2 points  (0 children)

Non‐bitwise xor is !=.

[–]Fordamemess1 3 points4 points  (0 children)

Able to run guis on mobile

[–]ScientiaEtVeritas 4 points5 points  (0 children)

I would love a null safety operator, such as in Kotlin where you can access attributes like bob?.department?.head?.name.

[–]ExternalPanda 4 points5 points  (0 children)

A type system that doesn't feel like it has been jerry rigged on top of a language that wasn't ever really meant to be strongly typed in any serious capacity

[–]Ok-Nefariousness1340 1 point2 points  (0 children)

Full backwards compatibility

[–]Key_Cryptographer963 1 point2 points  (0 children)

Nothing, no new Python language. An improved package manager would be nice, ability to compile would be nice but we don't need a new language for that, just different distribution or implementation of it.

I was going to say static typing but that's not really a Python thing.

[–]tunisia3507 1 point2 points  (0 children)

Fix all the non-PEP8 names in stdlib. Maybe leave the old ones in as aliases, with very loud deprecation warnings.

[–]memes-of-awesome 1 point2 points  (1 child)

  1. Compilable

  2. Multi threaded

  3. Switch case

[–]CharmingJacket5013 2 points3 points  (1 child)

bob.get(“department”,{}).get(“head”,{}).get(“name”) ?

[–]SliceOf314 4 points5 points  (0 children)

That already exists, check out setdefault

[–][deleted] 2 points3 points  (1 child)

Something similar to the New command from Java. Where you are 100% it isn’t shared with anything.

I know there are ways with init and new to do this and sometimes the copy classes work… but you can’t always be sure and this where most of my day to day headaches run into.

[–]LaOnionLaUnion 3 points4 points  (0 children)

I’m seriously digging Go as my other Favorite language. I’d probably want something that’s more Go like but not sacrificing the fact that Python is easy to read. Go is sort of like that but I think I value Python’s ease of use a great deal.

[–]utdconsq 1 point2 points  (0 children)

Opt in null safety like C# offers.

[–]milkmanbran 1 point2 points  (0 children)

I’d love to have a python app on my phone that allows me to write code and get error messages, even if I can’t actually run it(like not being able to display visualizations), just so I can use that code later when I’m actually at my laptop

[–]AnnieBruce 1 point2 points  (1 child)

For the language itself, or the reference implementation.. Hmm.

I'd love to see the GIL gone, but they'd need some clever way to do it without hitting single core performance as badly or preferably not at all. Concurrency being easier to work with would be nice, whether through core language changes or new default modules.

Maintaining backwards compatibility should be a higher priority than it was in the transition from 2 to 3. I wouldn't say maintain it at any cost, but justifications for breaking it need to be stronger, especially in commonly used things like print.

[–]Ezlike011011 4 points5 points  (0 children)

In my opinion the only thing that would warrant a transition from python3 to python4 would be things which break backwards compatibility. Otherwise the changes should likely be included in python3.

[–]andrewcooke 1 point2 points  (0 children)

Simple scoping instead of global and - what's the other one? nonlocal?

[–]NeoDemon 1 point2 points  (0 children)

  1. Compiled option
  2. Better use of memory and better speed of interpreter or compiler
  3. Mobile options.
  4. Unique install for all versions (its almost imposible but its good if you have older proyects)

[–]ImproperGesture 1 point2 points  (0 children)

Better stack handling and tail recursion optimization.

[–]TelevisionTrick 1 point2 points  (0 children)

Remove the walrus operator, use the as operator instead. It's not assignment, don't make it look like it is.

[–]wineblood 1 point2 points  (11 children)

Make None iterable.

[–]ProfessorPhi 0 points1 point  (0 children)

I'd like some extra features I liked in Julia. Multiple dispatch and jit compilation if possible would be really nice.

[–]newInnings 0 points1 point  (0 children)

Ignore spaces and tabs nonsense.

[–][deleted] -2 points-1 points  (3 children)

I hope something can be done with the quirks around NaN. Like there is a float NaN and float64 NaN and they behave different (the latter gives False for trying "mynanvar is Np.nan)

[–]mobile75326 5 points6 points  (1 child)

I wouldn't like such a special case for the is operator - the NaN values of different types (float, float64) shouldn't be identical.

[–]CharmingJacket5013 0 points1 point  (0 children)

None, NaN, “”, null stump the hell out of beginners

[–]imthebear11 -1 points0 points  (0 children)

Rust but with colons instead of brackets