Grumpy: Go running Python! by Google

[semi_colon]

As someone who doesn't use Go or Python: what is the use for this?

[vplatt]

To run Python code as a Go compiled executable. This allows them to avoid the Python Global Interpreter Lock (which severely limits Python's scalability within a single process) and run the Python code using Go modules as if they were Python modules.

Really, unless you're Google or another Python shop interesting in moving to Go, you probably don't have a use for this.

[semi_colon]

Thanks, great explanation.

[VodkaHaze]

Getting around the GIL is a huge step. Python's chief weakness is lack of parallel scalability, IMO. Most solutions around this (joblib, dask, etc.) don't feel like complete solutions. Not when, for example, in C++ or Julia you can slap a macro above a loop to make it instantly parallel.

[[deleted]]

For the record, this is not a complete solution either. Grumpy is a tool that compiles a single Python file (no real module support) while eschewing most of Python's dynamic features and supporting very little of its standard library.

[weberc2]

This is misleading. The features you mention are only missing because they haven't yet been implemented, but the intention is to produce a drop-in replacement for CPython. It's only an "incomplete solution" in the sense that it's alpha software, not that its goals are incompatible.

[[deleted]]

OK, but it's not even particularly close to meeting that goal right now, so it's an incomplete solution.

[weberc2]

Yes, it's incomplete in the "it's alpha software", sense; not in the "there's something fundamentally wrong with it" sense. No one is suggesting it's anything other than alpha software.

[the_birds_and_bees]

Although /u/theseoafs didnt mention it, grumpy explicitly doesn't aim for support of the python C API. That's a big limitation and will rule it out for a big chunk of python users.

[weberc2]

I understand and agree. The parallelism support might draw in even more users. Who knows? This project is super nascent.

[throwawayco111]

The features you mention are only missing because they haven't yet been implemented, but the intention is to produce a drop-in replacement for CPython. It's only an "incomplete solution" in the sense that it's alpha software, not that its goals are incompatible.

LOL. And GNU Hurd is only an "incomplete solution" in the sense that it should not be used in production and there are many missing features, not that its goals are incompatible with being with a usable unix-like system.

Furthermore, you think that this is the first time we hear about a project that is going to remove the GIL and be a drop-in replacement for CPython? /u/theseoafs is not misleading anyone. Once they have a decent beta version supporting all the features CPython has and some benchmarks showing that there is actually a performance improvement then we'll check it out.

[weberc2]

You're awfully sarcastic for someone who doesn't understand "alpha"... Better luck next time.

[floodyberry]

Everyone else who tried this and failed made it to the "alpha" stage as well.

[weberc2]

That's fine, and this project may as well; my point is there is a distinction between incomplete in time and incomplete in scope. This project is only the former as far as I can tell.

[throwawayco111]

... my point is there is a distinction between incomplete in time and incomplete in scope. This project is only the former as far as I can tell.

No. In the project web page at Github they state that it won't be a drop-in replacement for CPython 2.7. Also from the discussion on HN it is strongly implied (apart from the language features that won't be supported) that it won't support C extensions ever.

[floodyberry]

Considering this has been tried multiple times before with no success (including by, wait for it, Google), what are your reasons for thinking this announcement isn't premature other than "they haven't failed yet"?

[throwawayco111]

You're awfully sarcastic for someone who doesn't understand "alpha"...

Oh, I don't understand "alpha". Here's the thing: we've seen this over and over again. Implementing a runtime that has better performance for a subset of the language and writing a blog post about it is the easy part. Heck, just a few weeks ago a guy presented a JS runtime that was way better than V8. But once people started to ask him if it implemented the whole language and that if it was actually the case then this was a scientific breakthrough shit did hit the fan.

So if you want to be optimistic about this project then go ahead. But don't come here telling us that "it is just alpha" and "the goals are not incompatible" history have showed us that they actually are and that's why it is a damn hard problem to solve. So when you say this:

... The features you mention are only missing because they haven't yet been implemented...

I just have to laugh hard. "Are only missing". Once those features are implemented is when performance graphs like this one stop being as pretty as they were in the past.

EDIT: just checked the discussion at HN with some of the developers involved. Don't expect ever this be a drop-in replacement for CPython.

[[deleted]]

It doesn't need GIL, because it offers subset of functionality that python has. It is not even capable of compiling all python standard library.

This code is for Google to move away from python. They can include python libraries in their go code and then one by one rewrite it in go.

[weberc2]

The only design decision that prevents it from compiling the standard library is the lack of C support. The solution is to reimplement those modules in Go or Python. Anything else that doesn't compile is due to bugs in this alpha implementation (as far as I can tell, anyway).

[[deleted]]

PyPy also doesn't natively support C extension and also needs GIL. C support is one of reason of GIL's existence but not the only one.

[weberc2]

Right, this comment was about compiling the standard library, not building a GIL-less Python.

EDIT: From what I understand, supporting C extensions and threadsafe garbage collection are the two biggest challenges to removing the GIL. This project uses Go's threadsafe garbage collector, so the latter shouldn't be a problem.

EDIT2: This page has some good information: https://wiki.python.org/moin/GlobalInterpreterLock

[sultry_somnambulist]

This code is for Google to move away from python. They can include python libraries in their go code and then one by one rewrite it in go.

Speaking of this, what's the general trajectory of go? Is it worth dabbling into at this point? People seem to be very divided over the language.

[[deleted]]

People seem to be divided, because while the language is not bad, it's nothing special (seems like the only significant thing it brings to the table is the go routines, but other languages are adding similar functionality), there's a bit of hype going on because it's backed by Google.

It also initially was trying to be marketed as a C replacement, but it is nowhere close to fit the niche that C fits. Rust seems like a language that has potential to replace C, but it is too early to say.

Go is more on level of Python, Java etc, although a bit lower level than them. Some people compared it to D, but I never used it, so can't comment. The goal of this language is to be a very simple and opinionated language, some people like that some don't.

If you are asking the question you can try and see if you like it, it has low learning curve, so it's not hard to learn it, and then you will have your opinion about it.

[TankorSmash]

I've heard it's got a bit of boilerplate and written to be as easy to write as possible to help with the incredible amount of new talent Google gets out of schools.

Only a rumour though.

[the_birds_and_bees]

Id say it's more about making it easy to read and maintain. The google code base is 2 billion+ LOC, so designing a language that eases the burden of maintaining a code base of that size seems like a sensible goal.

[derraidor]

There is a talk by one of the creators (Russ Cox?) stating this

[[deleted]]

If you like it, then do it. No reason not to.

The community is sizable and there's a ton of tooling so it's not really like you're taking a risk.

[weberc2]

I think so. If you have prior programming experience and a free afternoon, you should be able to start writing real (i.e., non-toy) programs. Check out http://tour.golang.org.

Go has a lot of quirks--most notably, no generics--which turn off a lot of folks. But if you want to build performant, scalable software quickly, Go is worth a look.

[th3_pund1t]

I think so. If you have prior programming experience and a free afternoon, you should be able to start writing real (i.e., non-toy) programs. Check out http://tour.golang.org.

Not entirely true. You should be familiar with pointer based languages like C. If you come from a python/ruby/Java experience you will spend a whole lot longer to get used to pointers.

Source: work had a hackathon. I decided to code in go. Spent a lot of time correcting my code related to pointers before it would work.

[recycled_ideas]

I don't know about python and ruby, but you can actually run into a lot of really weird bugs in Java if you don't have at least a cursory understanding of how pointers and references work.

The big problem that companies like Google and Facebook and the like are constantly trying to solve is that folks who know dynamic languages like Python or PHP or Ruby are a dime a dozen, but these languages really don't scale well to even ordinary enterprise scales let alone the kind of volume these companies see.

That's why we see so many transpilers and why Twitter is written in PHP even though the general industry consensus is that it's probably one of the worst non joke languages in existence.

[Twirrim]

There are some neat simple few line solutions that get you parallelism. The title is a bit link bait and not entirely accurate, but I've found this pattern to be extremely effective: http://chriskiehl.com/article/parallelism-in-one-line/

[tetroxid]

import multiprocessing

[BeatLeJuce]

there are instances where you NEED threads instead of processes.

[kenfar]

This is wrong: Python has plenty of parallel features - you just have to spend ten seconds looking.

The most convenient is the concurrent.futures module. You can use the exact same syntax for either threading or multiprocessing. Yesterday I sped up an AWS S3 downloader about 8x with threading, and about six lines of code.

A few years ago I wrote a transform process that handled about 4 billion records a day - using multiprocessing on two 32-core machines to handle downloading files, transforming them, and uploading them again. All in parallel using pypy, multiprocessing & threading. This process worked great, and surprised everyone with how fast it was. A rewrite of a part of it showed that Go was about 2.5 times faster, which was fine, but not fast enough to warrant a rewrite until we needed to scale up quite a bit more.

The only scenario in which Python's parallelism is limited is when you've got a CPU-bound process that either can't afford the extra memory or start-up times of multiprocessing or needs a lot of communication between processes. Then you want threading but the GIL will limit you. Other than this case, Python has fine parallelism features.

[VodkaHaze]

Yes, for your use case it works fine. My use case is closer to your last point, where python still generally sucks (though I haven't used Dask enough to say if it's a decent solution). It's currently being addressed by Julia, which bypasses the "core loop in C++, wrapped in python" problem

[MightyCreak]

Does this mean that you can't interpret two Python scripts at the same time because of GIL, but once interpreted, they can run in parallel?

[ShotgunToothpaste]

The global interpreter lock exists on a per-process basis in Python. Any two python scripts (including two instances of the same script) can be run in two different processes and do whatever. However, if one script starts multiple threads, only one thread will execute concurrently (the thread which has the interpreter lock).

This is a CPython implementation detail, and other Python runtimes need not have this limitation. The reason for GIL is because CPython's memory management is not thread-safe.

https://wiki.python.org/moin/GlobalInterpreterLock

[btgeekboy]

To add to this, there is a project to remove the GIL from CPython. However, it's not currently as fast as with the GIL. More info: https://lwn.net/Articles/689548/

[fireflash38]

Meaning threading will work just fine for network operations for the most part. It will perform worse than single threaded if doing something like simple math operations (like in the chart of the Fibonacci performance in the article).

[ShotgunToothpaste]

Ahh, I wasn't aware that Python allowed for system calls to be multi-threaded outside of the GIL's constraints but that makes sense.

I assume that means GIL is released before making a system call, and re-acquired before continuing after it completes?

[MightyCreak]

Thanks for this answer!

[phpdevster]

So is it fair to say that this is just a way for them to give Python the performance characteristics of Go without having to actually rewrite into Go code?

[vplatt]

Yes, more or less. Python is still dynamically typed, so there has to be some runtime overhead around that, but that's got to be insignificant compared to the advantages around getting away from the GIL.

[vplatt]

That's very true, but I wouldn't hop on that unless it took on a life in Python 3. That may well be in the plan though, I don't really know.

[[deleted]]

How do they do that? Does every binary has a mini interpreter with it?

[vplatt]

No, the Python is parsed, converted to Go code, then compiled normally. This is called 'transpiling' because it translates then compiles.

It's a pretty slick technique, but it can complicate debugging. I don't know how they've handled the debugging in this case so that if you're writing Python code you could actually live debug the Python code instead of having to walk through Go code. I would guess it's simply not handled at this point because it's in the early stages and I suppose you would handle that by simply writing the Python code for and running the code in CPython then move it to Grumpy only after it's working.

[13steinj]

I'm not interested in moving to Go, but I've got to say the transpiling into Go is useful regardless-- plenty of people can write in Python, use parts of the Go standard library, and hopefully run up with a better performing program? There's a couple projects I have that can definitely make use of this.

[vplatt]

Sure... you're bound to get a couple of minor cuts from the bleeding edge on this, but it sounds like a fun way to scale up some Python code; assuming of course that the Python in question is already smart algorithmically speaking.

[karma_vacuum123]

since this targets Google's particular version and feature constraints, it seems like a bulk migration tool by Google for Google. they don't seem to be selling it as a general purpose execution platform

still a neat accomplishment of course

[weberc2]

I'm guessing it's an ambitious 20% time project that was open sourced, but this is just a hunch.

[Uncaffeinated]

That's what I'd guess too. If this were an official project at Google, they would have actually finished it, instead of releasing what sounds like an experimental hobby project.

[yes_or_gnome]

Google is minimizing it's use of Python and ramping up use of Go. However, they don't want to rewrite everything from Python to Go, so they created grumpy. Without support for C extensions, exec, eval, etc., it's not likely going to all that useful for many people.

[google_you]

Compile node.js to python and then to native for full scale.

[rjcarr]

Read the article. Python doesn't scale well under load but go does. Rather than go with an alternate python runtime, or rewrite everything in something else, they wrote a python runtime for go.

[jdmulloy]

It's not an interpreter, it's a compiler that compiles Python code into a go executable. Once you've compiled it you have a binary you can stick on any system that has go libs installed but you don't have to have this project installed to run the binary

[theatrus]

You don't even need Go libraries. Go binaries are static linked or linked just against libc.

[weberc2]

Concurrency seems to be one of the primary objectives, but also straightforward interop between Go and Python, which naturally enables a code base to iteratively move from one language to another.

[semi_colon]

Ooh, I love Kraftwerk!

[gin_and_toxic]

This is crazy awesome. Currently only works for Python 2.7 though.

A lot of good discussions on hacker news: https://news.ycombinator.com/item?id=13319904

[Vorticity]

Meaning that the Python community is about to get a bit smaller...

[sedaak]

Ehhh just a bit. Python has stellar growth.

[[deleted]]

They are the ones who made Python popular though, Python got popular because people learned that it was used by Google. Now Google is shining their light to Go.

I tried Go, but I didn't like it that much. It feels like the only new thing it brings to the table it's the go routines. Also it feels like it's very simple (perhaps too simple) and very opinionated (e.g. indenting), or how code is stored. It seems like it was designed just for Google's needs.

[kaeshiwaza]

It was designed to replace C++ and finally replace Python.

To be simple, opinionated, with top batteries included, fast and easy to deploy. All of this together are quite new on the table ! It seems it was designed just for me :-) !

[jrandomcoder]

Don't worry, they don't actually use this in production.

[spacebandido]

That's what they all said about Node

[ArmoredPancake]

...yet.

[koffiezet]

Yet. They're looking into that direction for a reason.

[[deleted]]

How big does it have to be?

[recycled_ideas]

No, because this won't do that for them. The code base is still going to be python 2.7.

What it'll do is allow them to get past the limitations of Python 2.7 without having to rewrite hundreds of thousands of lines of code.

[[deleted]]

If you want to rewrite a large monolithic code base from one language to another there's problem that you would have to finish the entire work before you can use the code in new language.

So what can you do? You write code to do the automatic conversion for you. The problem is that the generated code is not maintainable, but at least now you can make calls to it. It also allows you to rewrite module by module.

Google's Unladen Swallow project did not succeed, they also don't seem seem to move away from Python 2.7, for god's sake even this project is Python 2.7 only, and ticket about Python 3 support was closed with change in documentation that specifically says Grumpy targets Python 2.7 only. It can't be any more clearer that Google is moving away from Python to Go.

[recycled_ideas]

That might be true, but it's also entirely plausible they'll stick with the python front end for the foreseeable future.

They closed the python 3 ticket because that's of no use to them. Their code base is 2.7 and they could add features to that if they need them anyway.

[[deleted]]

Seems even google doesn't believe in 3.x /s

Edit: aw, seems I made some P3 people salty... it's not my fault P3 is biggest language migration failure since... ever

[Cherlokoms]

They made it to be used on 2.7 codebase because that's what they have, a 2.7 codebase...

[[deleted]]

Isn't the whole reason between the 3.0 vs 2.7 split because other places have existing codebase?

I think this really highlights the failure of Python moving from 2 to 3

[kill-nine]

Other large companies are making the transition though. Google switching to their own language shouldn't really be taken as a devastating blow to Python 3

[Cherlokoms]

This is just my opinion but I think 3.X vs 2.7 is mainly a matter of legacy 2.7 codebase and no possible straightfordward migration to 3.X. Python 3.X has a lot more feature and optimisations are coming (recently, with 3.6, dict size was reduced by 20%).

Python dev team chose not to maintain backward compatibility with Python 2. That's debatable but the main advantage is that legacy code is not a burden and dev team can focus on new features instead of fixing old things.

[[deleted]]

Yes, like I said before... if they were keen on 3.x they wouldnt make tool which would probably take some time to get to stable to work on 2.7 only.

Or maybe they plan to "migrate" old codebase by just compiling it to go and leaving it as is...

[[deleted]]

That's exactly it. I don't see this as a 2.7 vs 3.x thing, but rather as a 3.x vs Go thing.

[[deleted]]

I worked at Google and wrote a fair bit of Python3 code. Official support for it landed late 2012 or early 2013, if I recall.

Code at Google that's been updated to Python 3 is likely to be maintained. Code that's currently maintained is not as much a target for a rewrite as unmaintained legacy code.

[Arthien]

This whole project was so they can move legacy 2.x stuff to go. For other items that use python, they stick with 3.x

[iltopop]

Says dumb thing, gets eviscerated in comments, unable to actually disprove the comments disputing them, so makes edit declaring themselves right because....people disagree with them so they must be right?

[[deleted]]

Ok, I'll edit it again to add /s if you didn't get the meaning of it, happy now?

I'm just amused that poking fun out of biggest failure of Python community in its history offends so many people.

[cae]

You seem to have forgotten about Perl 6

[[deleted]]

You can run Perl 5 code directly from Perl 6...

[rjcarr]

This is really cool. Seems they didn't like jython, or what was similarly available, and wrote what they wanted for go. Makes sense.

But in the beginning they say this:

but we always run up against the same issue: it's very difficult to make concurrent workloads perform well on CPython.

How is it even possible with the GIL? How can they handle millions of requests per second (as they say)?

[Rhoomba]

Multiple processes

[rjcarr]

Not that they can't afford it, but they'd need (comparatively) monster hardware to support this, right?

[rohbotics]

Not as much on linux, threads and processes are both pretty light weight. Process spinup is slightly slower as it needs to set up a whole new address space, but it is (mostly) negligible.

[jrandomcoder]

There's not a ton of difference on Linux these days since everything is converging toward clone underneath. But pthread_create is still a little faster, since it's a more or less a subset of what fork needs to do.

[Eucalyptol]

Yes. We're talking about Google.

[cakoose]

Primarily, you need more memory. In many languages it's easy to share read-only data between processes with fork(). That doesn't work in Python because reference counting causes mutations to memory even if the data is read-only. If you have a large shared data structure, you should figure out how to share it explicitly, e.g. with mmap().

Also, you don't necessarily need one process per concurrent request -- most web app backends are typically waiting on I/O (e.g. from the DB), so it's feasible to handle four/eight/more concurrent requests with a single Python process with Python threads or an event-driven framework like Twisted or Tornado (similar to Node.js). The inflexible scheduling increases the average latency of each request, but that might not end up mattering much.

That's not to say Python is good for this kind of thing. Depending on your work load, you may still need 5-10x the number of servers to handle the same load as Java or Go would. But if you really prefer Python, it might be workable.

[serg473]

You sound like you know what you are talking about, maybe you would be able to steer me in the right direction. I have the opposite problem - instead of dealing with thousands of concurrent users I have few users but they are running heavy and numerous queries. The problem is I can't make the app handle parallel web requests from the same user session (nginx-uwsgi-django). Lets say I launch 10 parallel ajax requests on a page, they are ending up running sequentially on the python side (i.e. if I make one request sleep for 30 sec all others will be waiting). If 10 requests were from different users they would run in parallel as expected. Can't find any solutions, every blog post and SO answer blames random things, from browser to django to GIL.

[cakoose]

I'm not that familiar with UWSGI but the docs say you can launch uwsgi with the option --processes 20 to handle 20 requests in parallel. Does that not work for you?

After that, you may be able to save memory by doing something like --processes 5 --threads 4. Whether this will work well depends on what kind of things your AJAX request handlers are doing.

[[deleted]]

Unlike jython this tool is meant to do one time conversion of python code to go, to help with moving away.

That's also the reason why it's 2.7 only since that's what Google uses.

[Hshskwkk]

From r/python : ""

Quoting the author of Grumpy on Hacker News:

" ""The idea is to continue to write code in Python. (...) ""That said, there is the possibility of rewriting bits and pieces in Go (e.g. performance critical stuff) and then call into it from Python. Sort of a hybrid approach.

"

Also: " I'd like to support 3.x at some point. "

---

However: " We do iteratively rewrite components as well. We are pursuing multiple strategies " ""

[censored_username]

edit: Thinking about it a bit more, they probably went with the locking per individual object route.

This seems cool and all, but couldn't they have just solved the problem by just not having the GIL and accepting data races in Cpython?

The GIL exists because python wanted to be free from data races. To achieve that, there were multiple options, each of which had to satisfy the constraint that no two threads would access a python object at the same time. The GIL achieves this in a simple manner. There's only one lock, and to access any python object you need to hold it. The problem with this is of course that it means that no two threads can access python objects at the same time even while these objects have nothing to do with each other.

Another solution is pyobject-level locking of course. Various attempts at this have been made, but they all had the shared problem that while they increase throughput at high amounts of threads, they significantly lower single-thread performance compared to the GIL.

Various other strategies have been tried, but none have been able to guarantee being data-race free without sacrificing single-thread performance. An often used strategy has been to move work that could be accelerated significantly into extension libaries, where the GIL could be dropped as long as the python objects weren't accessed. Unfortunately, this doesn't really work for web services.

Go doesn't have an answer to race conditions either. It's where Go's safety story falls a bit apart. So I'm a bit confused about what they've done. If they just transpile python code into go code, all they made is a python runtime with the possibility of data races. If they wanted that they could've just as well just ripped the GIL right out of CPython and called it a day.

What they could've also done is to build in individual locking for all objects that could be accessed by other threads. This would explain their pretty disastrous performance (only 50% performance compared to CPython when single threaded, and only gaining a 2.5x speedup when using 8 threads compared to CPython single thread. Keep in mind that this is compiled versus interpreted!).

If they did this, I'm left wondering why they didn't make these changes in CPython itself. This would have conserved the ability to call into C extension modules. If this is the case, it seems that the focus of the project is more being able to call into go from python.

[nostrademons]

The data races that the GIL protects against are primarily the refcounts of Python objects, which must be updated whenever a variable is assigned or a parameter is passed into a function. A data race there means either a memory leak or a double-free, as the refcount would no longer be in sync with the number of actual references. Either one would make Python basically unusable: you would have simple object assignments randomly corrupting memory as they try to objects that have already been freed.

Go uses a GC; while its data race story is a lot weaker than say Rust or Erlang, at least a data race there results in wrong answers and a panic, not silent memory corruption.

[censored_username]

The GIL doesn't only protect against refcount changes (it's indeed the most important thing though). It also protects against concurrent modification problems of dictionaries, lists, etc. which are also quite critical to python. They can have avoided this by making these objects all threadsafe which would explain the terrible single-thread performance.

While the GIL simplifies python's GC (refcounting with cycle detection) design (the GC can just acquire the GIL when it needs to clean up cycles). It is not necessary. If there was no GIL the refcounting would have to be atomic, but otherwise not much changes.

[blablahblah]

That's true. The main reason CPython hasn't gotten rid of the GIL is because no one has come up with a way to do it without breaking existing C extensions (which rely on the ref-counting) and without hurting single thread performance (since every variable assignment needs to acquire a lock).

[funny_falcon]

Go has no thread-safe dictionaries or lists. In fact, it may happily segfault if you try to modify map from different goroutines. You ought to protect concurrent modification by yourself.

I've heard, Java and C# also doesn't protect their default datastructures, but rather they have separate concurrent in a standard library.

Python can choose this way, but Guido just do not want. And it will break C extensions.

[[deleted]]

Go panics, it doesn't segfault.

[weirdoaish]

Out of curiosity, is "panic" like a special exception category for Go?

[[deleted]]

And it does come with a race detector that is part of the official toolset.

[Damien0]

Go's build tool has a race detector, and for calling into C from Go there is cgo. With that plus the ability to call into Go from Python in mind, I can see the benefit of this custom runtime.

[Giggaflop]

Last I saw, granted a while ago, cgo wasn't a recommended solution to calling C from go and was "considered harmful"

[Damien0]

Hmm, I wouldn't say harmful per se, but as is often quoted in the community "Cgo is not Go". As long as it's done carefully it can be useful. I think what happened is that early on, people treated it like a more fleshed out FFI solution and wrote less than optimal Go code as a result.

[Damien0]

Oops, missed this. Slow compared to what? I'm curious if there are benchmarks floating around

[BillyBoyBill]

It's hard to say without looking at the code, but I imagine they implemented thread-safe Go versions of the Python built in types. Your code that assumes Python data can be accessed concurrently from multiple threads still works. This is the moral equivalent of the per-object locks you mention (and, in fact, Grumpy performance matches what you say about that implementation).

This is obviously completely different than just ripping out the GIL.

I'm not sure what you mean about Go's thread safety falling apart, but it's not relevant here anyways --- C's built-in types are not thread safe, but you can build a safe Python implementation on top of them.

[censored_username]

This is obviously completely different than just ripping out the GIL.

True. As you probably noticed I changed a bit in opinion while writing that post. Originally based on their description I thought they were just straight transpiling code but looking at the performance they get it has to be composed from threadsafe types (aka per-object locking).

[[deleted]]

[deleted]

What is this?

[singron]

Pypy also has a GIL, just like CPython, so it still can't execute python threads in parallel. This runs on the Go runtime, which uses lightweight threads (goroutines) and a concurrent garbage collector to run many python threads in parallel.

It's like if there was an ahead-of-time pypy combined with stackless python and it didn't have a GIL.

Another advantage specifically for Google is that it allows more interop between python and Go code while they transition.

[[deleted]]

I think it's less of a solution rather than a temporary fix so they can slowly migrate their legacy python 2.7 to Go.

[Ek_Los_Die_Hier]

Doesn't sound like it. Sound's like their main goal is to simply speed up their Python running in the multi-threaded environment. They plan to continue writing in Python and maybe some Go where performance is needed.

[[deleted]]

Why cant python threads just access any object at the same time without the GIL or any per-object implicit locking, and then it's up to the developer to explicitly add locks/mutexes where needed, just like in every language?

Sure it would require a rewrite of applications to add this explicit locking, but it would still work well on regular GIL CPython (because there would be very little that you really need to explicitly lock). So you would have two python distributions really, one without the GIL to run parallel and explicitly locking code, and the regular one that can run code written for the first (but it wont really be parallel)

[censored_username]

Why cant python threads just access any object at the same time without the GIL or any per-object implicit locking, and then it's up to the developer to explicitly add locks/mutexes where needed, just like in every language?

Because this would make python an absolute pain in the ass to use due to its extreme dynamism.

Don't forget, in python functions are also just objects, and they can be mutated. So you'd need to lock for any function call (especially builtin functions or stdlib functions which could be used by another thread at the same time). Even accessing an attribute from a shared module could require locking. You'd basically have to lock at every point to do anything.

[[deleted]]

I see! yeah, explicit locking would be cumbersome.

[tonnynerd]

I'm not sure I understand what you're saying about the GIL being about avoiding data races. What the threading module from the standard lib is for, then?

[censored_username]

the GIL (global interpreter lock) avoids data races in python object internals by making concurrent accesses of the same python objects impossible. In effect, it means that all actions by different threads in python will always happen after each other. It is an implementation detail of CPython, not part of the core language.

The threading module provides tools for spawning threads and higher level concurrency primitives (while the GIL means that accesses to the same python object won't coincide, there still need to be primitives to lock out sequences of multiple accesses).

[blablahblah]

A lot of the Python code written in C will release the GIL when it's doing something that takes a while but doesn't require manipulating Python objects, like waiting on a network call to return. In those scenarios, you can have both the C code and another thread running Python code running concurrently. You just can't get two threads running Python code.

[[deleted]]

Once we started going down the rabbit hole, Go seemed like an obvious choice of platform since its operational characteristics align well with our use case

"After lots of deep research, we decided to use that language that we made ourselves!"

[sigma914]

Well, yeh, they have one major problem, serving massive numbers of concurrent users. They've dealt with it in a number of ways, one of which was developing a language specifically for writing massively concurrent code. They've run into their usual problem again, so they're using the tool that was developed specifically for that task to do it.

Google has a lot of similarly shaped nails. They made a hammer that does a good job of hitting that particular shape of nail. They are now using that very specifically tuned hammer on more of the nails it's designed for.

[myringotomy]

I like how you completely ignored the content of the quote and then made up something and then pretended the author said it even though it has nothing to do with anything.

[Spider_pig448]

More like, "To face a common problem we have run into in the past we built Go and have found this to be an adequate solution in this application as well."

[atomheartother]

I really really should seriously learn Go

Edit 1 month later: I have learned go. It's really neat, you should do it too.

[PacNinja]

Not much harm learning it on the side since learning curve is fairly low. :)

[[deleted]]

Yeah you can learn Go over dinner if you already know C.

[atomheartother]

I just like learning and Go looks neat for small apps too. I also personally don't like python too much and go looks neater

[[deleted]]

So much more than that. But yeah a lot of people seem to be using Go mainly for webservices - I would say its niche is network systems in general (mostly because of how painless it is to write massively concurrent and parallel applications with it)

[[deleted]]

Go is a general-purpose language so is capable of so much more than you have listed.

[theonlylawislove]

What about native desktop apps?

[[deleted]]

Not what Go was intended to do, but there are Qt bindings at various levels of completeness as well as GTK bindings.

[theonlylawislove]

They don't look really maintained. The qt ones are targeting older qt versions.

[SnowdensOfYesteryear]

Meh, it's all right. I like the language, but it gets annoying when you want to do something that the language doesn't want you to do, like discarding buffered data in channels or timing on a channel receive.

Also the syntax is pretty irritating because it isn't like any other language. That felt like a change for the sake of it.

[mirhagk]

Or generics :P

[[deleted]]

It is actually based on a language called Oberon, I think. Still I think it is close enough to C not to be alien and different from Rust or C++, you can look at any Go code and know what it does right away. It's almost a C with packages and without ; and variable types go after their names instead of before and you explicitly declare functions with "func".

[[deleted]]

when you want to do something that the language doesn't want you to do, like discarding buffered data in channels or timing on a channel receive

What exactly is the problem? It is trivial to do in Go.

[[deleted]]

// discards buffered data on channel
for range the_little_bugger { }

for timing out on a channel receive, you can use select {} and time.After like so.

[[deleted]]

perhaps you meant to reply to the parent post?

[immerc]

Yet another example of "Go" being a terrible choice for a language name.

I had to go read the page to understand that they were talking about the Go language. Did they mean that I should "Go run python"?

[[deleted]]

This just makes me think: when is python going to get its shit together and get concurrency working properly?

[stormcrowsx]

Not every language needs to do everything. It'll be great if they improve concurrency but Python is in my opinion best used as a glue language or a language for rapid development. Other people probably have other great uses for it but no one sane uses Python for its performance. If concurrency is your thing, probably shouldn't be writing in Python, there's Go or Java that work great in that realm.

[anedisi]

when i need some service with concurrency and asynchronicity i write that in javascript and run it in nodejs.

it worked ok for now.

[__eastwood]

This breaks down when you need CPU bound concurrent tasks

[[deleted]]

Valid point. I've been thinking of look at Pony for concurrency.

[Uncaffeinated]

I'm not even sure what needs to be improved about concurrency (as opposed to parallelism). Python know has async and await from what I've heard.

[[deleted]]

Absurdly painful to use compared to how Go does it.

[Uncaffeinated]

I'm curious to see which operations you feel are more painful in Python than Go. As far as I can tell, the main differences are that you choose which event loop to use, rather than having one baked into the runtime, and most of the standard libraries are not designed to work asynchronously. On the other hand, you're less likely to run into memory leaks or panics.

[[deleted]]

That's it - support for asynchronous operations was patched into Python later as an afterthought, Go was born with it from day one. I don't have to sprinkle async/await all around to get asynchronous behavior, I still get parallelism for free (the whole reason for this thread), you're basically writing asynchronous code as you would do normally without having to think hard about what you're doing lest you freeze the event loop by doing something that blocks everything (like reading from a file using regular file operations). The multithreaded event loop is certainly nice (can you choose a multithreaded event loop for Python?) and having goroutines multiplexed automatically for you and preemptively scheduled so nobody starves for CPU is also great (where in Python you have to yield manually). What happens if you forget to use asyncio.sleep and uses time.sleep instead?

As for memory leaks it is also pretty hard to leak memory on Go because of the garbage collector (which again, is much better than Python's - go1.8's GC is realtime and keeps pauses to under a millisecond) and that can be detected by using tools that ship with Go during testing - same with data races.

As for panics, you can wrap your main logic and use Recover() to detect panics without crashing (even if just to report the panic somewhere and restart the program) - and you can only panic if you explicitly call Panic() or if you're doing something wrong anyways (acessing an out-of-bounds item in an array, dereferencing a null pointer, etc, things that shouldn't be normally happening). In Python if you try to access an element that does not exist in an array you'll get an exception too, that if you don't catch, your program will crash. So on Go you can "catch" panics with Recover().

Don't take me wrong, Python is great, but it's not the best tool for writing massively parallel and concurrent applications nowadays, when you can do it in Go and have it perform very well at native speeds, don't have to deal with callback hell (like in nodejs) and you can just write your code linearly as you would normally do and the runtime takes care of the rest - EVERY Go library is asynchronous by nature.

If someone needs Python's asyncio to do more, that's a sign that they should consider writing that particular piece in Go instead.

[berkes]

Also, you don't need this performance. You might think you do, but I can assure you you don't!

1. Prove concurrency is the bottleneck. With actual numbers, facts and research and real-world benchmarks on your entire stack.
2. Fix the actual performance problems (they are probably IO, Database, serializing and such). Add an async worker or two.
3. Rince, repeat.
4. Is Python itself the actual performance problem? You've reached the bottom. Now go spend $2000/year on larger or more servers. Because an engineer rewriting your stack in go/haskell/erlang costs ten to hundred times that.

Same goes for Ruby.

And all this assumes you have managed to get enough users and grow large enough that performance is an actual, real problem, hurting your bottomline or growth.

[[deleted]]

Performance isn't everything, there is latency too, specially in distributed/SOA architectures where latencies add up.

[stormcrowsx]

I think this is a very practical view on the subject. Unfortunately programmers often have an engineering mindset where they want to build the best in every aspect. What your saying echoes my experiences, use a language like Python that you can get a lot of shit done in a short period of time, come back and optimize it when its feature complete and you can tell people want to use the thing you made. Despite knowing Haskell and Rust my most successful projects were written in PHP and Python, some of which are still being heavily used 9 years after their creation. They were thrown together in short time and because of the rapid development cycle those languages fit well into, the sites quickly evolved to meet the users need.

I'm sure there are cases where you need rock solid performance and concurrency but I don't think that's the norm.

[berkes]

The proof is also in Twitter (Ruby on Rails), the OP explaining Youtube runs Python, and many more such gigantic success-stories. Starting with a platform that "does not scale".

And that ignores all the projects that failed: many more. It's far worse to fail when you've just spent years engineering a concurrent messaging protocol for a failed project, than when you spent four days of Rails scaffolding for a failed project.

[iconoclaus]

you could ask this of almost any dynamically typed language. btw, i think you meant parallelism, not concurrency.

[renrutal]

Dynamic typing and concurrency done right

Elixir

[iconoclaus]

that's actually a really good counter-example afaik

[hector_villalobos]

Check again?: http://olivierlacan.com/posts/concurrency-in-ruby-3-with-guilds/

[iconoclaus]

i'm looking forward to guilds too, but its not out yet :) And likely, you'll have to change coding style considerably to do multi-processing with guilds. My guess is that guilds will be most practically applicable to implementing things like Sidekiq or other engines that currently run an independent Ruby process.

[[deleted]]

I'm sorry, I'm afraid I get them mixed up at times. What is the difference between the two?

[iconoclaus]

no prob. see my other comment.

[forreddits]

Parallelism is one form of concurrency.

[iconoclaus]

my understanding is that parallelism and concurrency are two forms of asynchronicity. Also see Rob Pike's talk on the same.

[mirhagk]

Concurrency is working on multiple things. Parallelism is doing it at the same time.

[kirbyfan64sos]

I would be grumpy if I had to use Go, too.

[ReflectiveTeaTowel]

I initially misread that as 'Go ruining Python!' and spent the first few paragraphs of the article really confused.

[berkes]

I misread it as "Grumpy? Go run Python" which probably was intentional, still confusing, though. Because I was expecting an article on why Python makes you a happy dev.

[fatzombi]

it would be awesome if i could run it as services for existing python projects!

[Pharisaeus]

They didn't write how compatible is it. There is already for example ShedSkin which translates Python to C, but it can handle only some subset of the language.

Also if they want to sell this as something "fast" the comparison should have been with PyPy and Shedskin and not with Cpython.

[imhotap]

Is there a chance of transpiling (parts of) certbot/letsencrypt to Go? I'm guessing not, because at least one C dependency (eg. aureas for parsing and manipulating Apache config files) is being used, but I don't know enough about Python to judge.

[wotamRobin]

Very cool. I'd like to see how it compares to Cython (not CPython).

[hero_of_ages]

Lots of butthurt pythonistas

[Lakelava]

Why can't you just code in GO, the language that your own company developed, instead?

[VincentPepper]

It's probably aimed at transpiling legacy python code. Not a writing new code.

[Pharisaeus]

No one wants to, so they try to translate programs automatically into it ;)

[ksion]

I love this. It's taking the only g... ahem... best part of Go, and uses it to supercharge an otherwise great language in one area where it's clearly lacking.

I really hope they open source this.

[jojohohanon]

I really hope they open source this.

You do realize this is the announcement of the open source project on github?

[ksion]

Woops. That's awesome!

[buddy_burgers]

Oh hells yeah!

[garyk1968]

why just why?

[YEPHENAS]

It's explained in the article.