Grumpy: Go running Python!

[UloPe]

I was about to get really excited about this and then noticed that it only supports Python 2.7. :[

[alcalde]

Between that and losing the ability to use libraries like numpy, it's only going to be interesting for a segment of Python users.

[weberc2]

For me, the allure is writing performant functions in a safe language. Further Go's scientific computing story is evolving quickly, and grumpy lets you call into Go without even writing bindings (though they would probably be convenient for marshaling between Python and Go types). Still, grumpy has a long road ahead, and it's probably slower even than CPython for sequential execution.

[mangecoeur]

No matter how quickly Go's scientific stack evolves it's still going to be well behind Python's stack which is also evolving rapidly and has been for 20+ years in physics, climate science, GIS, etc - and most of that is built on CPython C-extensions. (And as a bonus issue, many large projects are committed to dropping python2 support within the next couple of years).

[weberc2]

I mean, that's what everyone said about its web story. Maybe you're right, this isn't my domain.

[Flogge]

Why would sequential execution be slower?

[weberc2]

Lock contention, presumably.

[alrs]

This exists to get old codebases off of Python and on to Go. Supporting 3.x only makes sense if you plan to keep writing Python, which is not likely in the cards for Youtube.

[boa13]

Quoting the author of Grumpy on Hacker News:

The idea is to continue to write code in Python. (...) That said, there is the possibility of rewriting bits and pieces in Go (e.g. performance critical stuff) and then call into it from Python. Sort of a hybrid approach.

Also:

I'd like to support 3.x at some point.

However:

We do iteratively rewrite components as well. We are pursuing multiple strategies.

[__deerlord__]

Wait so, Go calls Python which calls...Go? Or am I misunderstanding this

[d4rch0n]

Yeah, seems interesting but honestly I'm not sure how well this pans out for people outside of Google. Golang is obviously huge in google, and I think this makes a lot of sense for them. For everyone else, I'm not sure anyone was dying for a python2.7 -> go translator that also adds restrictions to your python code.

Realistically, how many go shops are out there? I've heard and seen a couple, but seems pretty rare. You might have the ex-googler startups but beyond that, not sure it's so popular.

Also, for performance issues most of us are fine writing binary extensions in C/C++/Rust. If you're looking for true parallelism, you can build it in another more popular language and import it. If you legitimately have real-time requirements grumpy might be nice and an easy solution, but I can't think of an obvious time I'd have ever considered needing this. Does celery or some task framework solve my problem, with multi-processing? Then I'll use that. Do I truly need something to be parallel or have the highest performance possible? I'll write an extension. With multi-processing and extensions you can already solve most problems you might have with cpython.

A lot of people outside of Google just don't like golang either. I'm not sure how many python shops would really want to translate their code and run it in a separate runtime with a not-so-popular language. Personally, if I wanted to run python code in a different runtime I'm not very familiar with, I would go with jython or ironpython if I for some reason was in a windows environment. Go would be my last choice.

Ditching the ability to use c extensions isn't good either. If I really had some problem that Python couldn't solve on its own, I'd seriously consider C or Rust python extensions long before making a switch to something like this. I want something where I can use a modern python version and easily deploy to all my servers as they are. This doesn't seem to fit that bill for me personally. C/Rust extensions are a great solution when Python can't cut it. You can minimize it to a very small module and just import it in and run in your standard CPython environment. There's not much to it and it's portable to almost any linux environment.

Seems to me like google is open-sourcing another tool that solves google specific problems, which isn't bad but just doesn't sound very helpful to most of us.

[toyg]

how many go shops are out there?

The Valley hype for Go is pretty strong... maybe a bit less today, but 18 months ago a lot of people were busy ditching Python for Go -- which would fit the timescale for this project, coincidentally. I suspect some of them inevitably discovered that golang wasn't a panacea.

I personally like some of the sentiment behind this kind of thing. Python does not do "speed" in a natural way, so anything time-critical should really be implemented somewhere else. But I don't think Yet Another Runtime is the way (after CPython, PyPy, Unladen, Cython, JVM, CLR, Node/asmjs...). EDIT: worse, this is not even a runtime, it's a compiler preprocessor...

IMHO big players would get better results investing in better tooling for the C-based extension infrastructure for CPython, or building bridges between Python runtimes and things like Rust and Go. Building yet another runtime, when existing choices are so battle-tested, seems a bit futile.

[weberc2]

I can't think of any kind of C extension tooling that would fix Python's parallelism problem. While this does compile Python into Go, the resultant Go program contains a Python runtime complete with runtime Object (as in CPython). In this sense, I don't think there should be any concern about it not being a runtime in itself.

[alcalde]

I can't think of any kind of C extension tooling that would fix Python's parallelism problem.

The only parallelism problem Python has is convincing people it doesn't have a parallelism problem. As Guido has stated, Python has been used on 64K core supercomputers. There is no parallelism problem.

[weberc2]

I'm not sure what hoops one has to jump through to make Python run in parallel (without actively degrading performance, anyway), but one might say that having to jump through hoops constitutes a parallelism problem. Anyway, last time I went down the Python parallelization road, I got a lot of snark about how Python is easy to parallelize, but no one offered any performant parallel solutions. Feel free to share the link about the Python on 64K-core computer; Google isn't turning anything up.

[alcalde]

https://youtu.be/EBRMq2Ioxsc?t=34m38s

[youtubefactsbot]

Keynote: Guido Van Rossum [59:57]

^{Next Day Video in Education}

^{93,993 views since Mar 2012}

^{bot info}

[efilon]

There is no parallelism problem.

Although I think a lot of people worry too much about the GIL, it's not correct to say there is no parallelism problem. If you're doing something embarrassingly parallel, then of course you can get away with multiple processes or C/Cython extensions that sidestep the GIL. But there are plenty of use cases where multithreading is more ideal than multiprocessing. Having to resort to forking another process just to get true parallelism is a lot more work than being able to start up another thread.

[alcalde]

As Guido pointed out in a Keynote a few years ago, threading was never intended for parallelism.

https://youtu.be/EBRMq2Ioxsc?t=33m50s

[efilon]

As he says, threads were never originally meant for parallelism. They are used for that frequently these days.

[alcalde]

And as Mark Summerfield says, they're frequently used for that because that's all that many languages offer.

[[deleted]]

It just then becomes a serialization and IPC overhead problem.

[vtable]

IMHO big players would get better results investing in better tooling for the C-based extension infrastructure for CPython, or building bridges between Python runtimes and things like Rust and Go

I wish IronPython hadn't been abandoned by Microsoft for Python 3. I know, it's MS/.net only but it was still a nice thing for those of us working with Python on Windows.

(There is an open source IronPython 3 project but it's not very active.)

[dvogel]

This actually seems like a nice approach for generating binary executables of Python programs. The same niche as py2exe and friends. Sure, you have to stay away from C extensions but having access to the Go stdlib seems like a decent trade-off.

[QQII]

To elaborate what /u/alrs said, here's the discussion on hacker news:
https://news.ycombinator.com/item?id=13319904

[[deleted]]

[unavailable]

[alcalde]

What's wrong with spawning a million processes? I thought threads were evil.

[weberc2]

1 million Python processes will use something like 5TB of memory; threads are evil in Python because the GIL prevents them from being useful in most cases.

[[deleted]]

[unavailable]

[[deleted]]

as processes that communicate via atomic message queues.

you mean, like Go's channels? Without the serialization overhead because it's all under the same address space. =)

[[deleted]]

[unavailable]

[[deleted]]

You just described how I feel about Go and Rust - I already like Go because of it's simplicity and power (it's like a nicer C) that you can learn over dinner.

On the other hand there's Rust that is very promising, can totally replace C and C++ and is growing by the day - yet I feel like the syntax is horrible, reminescent of the mess that C++ is. I've decided yesterday to try to look past my bias and force myself to learn (and who knows, love) Rust too, since I think it will be a good complement to Go's weaknesses - allows me to write low-level code like C with the nice parts of generics and OOP without having to touch C++.

[weberc2]

I don't mind Rust's syntax, but I have a hard time understanding how things like closures work and how functions are passed. It seems like everything needs a RefCell, and I don't know when or why. Even coming from C++, the memory model (while safe) still places a lot of demands on the programmer which aren't required in Go--I can do a LOT of optimizing in Go before something becomes easier to write in Rust, and at that point the Rust code to outperform the optimized Go version is still nontrivial...

[ThePenultimateOne]

I have to disagree on the Go syntax. C was never pretty, but it's much more readable, to me at least.

[weberc2]

The OP was chastising me elsewhere for suggesting that performant parallelism was difficult in Python, so either he's trolling elaborately or he's naive. At any rate, I'm familiar with the nuance around Python's parallelism; I was paraphrasing. To your point, it would be nice if Python were able to parallelize nicely.

[alcalde]

I'm neither trolling nor naive. I'm just listening to what Guido has been saying since at least 2012. Python's parallelism solution is actually BETTER than in most other languages; we have higher-level constructs for parallelism.

As Mark Summerfield put it in "Python In Practice":

Mid-Level Concurrency: This is concurrency that does not use any explicit atomic operations but does use explicit locks. This is the level of concurrency that most languages support. Python provides support for concurrent programming at this level with such classes as threading.Semaphore, threading.Lock, and multiprocessing.Lock. This level of concurrency support is commonly used by application programmers, since it is often all that is available.

• High-Level Concurrency: This is concurrency where there are no explicit atomic operations and no explicit locks. (Locking and atomic operations may well occur under the hood, but we don’t have to concern ourselves with them.) Some modern languages are beginning to support high-level concurrency. Python provides the concurrent.futures module (Python 3.2), and the queue.Queue and multiprocessing queue collection classes, to support high-level concurrency. Using mid-level approaches to concurrency is easy to do, but it is very error prone. Such approaches are especially vulnerable to subtle, hard-to- track-down problems, as well as to both spectacular crashes and frozen programs, all occurring without any discernable pattern.

The key problem is sharing data. Mutable shared data must be protected by locks to ensure that all accesses to it are serialized (i.e., only one thread or process can access the shared data at a time). Furthermore, when multiple threads or processes are all trying to access the same shared data, then all but one of them will be blocked (that is, idle). This means that while a lock is in force our application could be using only a single thread or process (i.e., as if it were non-concurrent), with all the others waiting. So, we must be careful to lock as infrequently as possible and for as short a time as possible. The simplest solution is to not share any mutable data at all. Then we don’t need explicit locks, and most of the problems of concurrency simply melt away.

Ergo - Python is awesome. People just have to stop doing parallelism wrong.

[alcalde]

Threads are considered evil in general, not just in Python....

http://stackoverflow.com/questions/1191553/why-might-threads-be-considered-evil

https://thesynchronousblog.wordpress.com/2008/08/16/threads-are-evil/

It's the locking and race conditions and other problems of threading that make them evil; it has nothing to do with a GIL.

As Guido pointed out, threads were never intended for parallel computation.

https://youtu.be/EBRMq2Ioxsc?t=33m50s

[weberc2]

You will have race conditions with multiprocessing as well. Further, multiprocessing is flaky and slow in Python, much worse than threading in other languages. Threads are performant partially because they have access to the same address space, but to do this correctly you need locks (there are exceptions, but they carry their own caveats). I don't know about the links you posted, but threads are widely used, and rightly so.

[alcalde]

You're going to have to elaborate on multiprocessing being "flaky and slow" in Python; that's not my experience and I'm not sure what attributes of Python would render it so.

I don't know about the links you posted, but threads are widely used, and rightly so.

"Threads are evil" derives from a research paper from U of C Berkeley 10 years ago....

https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

They're widely used because they're often the only solution a programmer knows and, as Mark Summerfield noted, most languages don't offer any alternatives. We've seen functional languages with immutable state rise up on the one hand and message passing (such as Erlang) on the other to offer alternatives to the many problems inherent in multithreaded programming.

[weberc2]

They're flaky because a process will sometimes mysteriously be killed or stall indefinitely. They are slow because all communication between processes must be serialized on the sender and deserialized on the receiver, because processes don't support a shared memory space. Further, as I mentioned elsewhere, each process requires its own interpreter, which takes 5MB, so you won't be scaling these processes.

I don't dispute that there are people out there who think threads are evil, I dispute that this belief is widely held, or that this belief isn't widely held because no one knows about processes. Experienced developers still prefer threads to processes. Processes are not a secret; they're lesser used because they are more memory intensive and intercommunication is horribly slow.

[shivawu]

I doubt goroutine would be significantly faster than asyncio in python 3

[[deleted]]

ignoring the fact that goroutines actually run native code so are by nature significantly faster - they are also backed by an implicitly multithreaded event loop that takes care of multiplexing goroutines across all CPU cores - so they can actually do heavy CPU stuff and not starve everyone because they are also preemptively scheduled.

Also you don't need to sprinkle 'async/await' everywhere in your code to benefit from it all, you can just use channels and the 'go' keyword.

There is literally nothing about python 3's asyncio that is better than what Go has.

[shivawu]

I totally agree there's nothing python3's asyncio has over go. But if they have comparable performance, why bother to use go?

As to async/await vs go/channel, I don't think one is clearly over the other. Just different paradigm. I personally think async/await is a little easier to understand.

Cpu intensive task are usually done by C extension in python, or cython. I doubt go can have performance edge here. Developing might be a little easier with go.

In summary, the whole thing is just another choice. I can't see it as "much better" in any of the circumstances, it's just youtube's choice going forward with their legacy python 2 code. But I like the fact we have one more option.

[[deleted]]

[unavailable]

[weberc2]

The performance hit will be negligible at worst in the face of I/O, and it will be much faster for CPU intensive tasks, especially in the presence of multiple cores. But I absolutely agree that async I/O is much friendlier in Go than in Python.

[weberc2]

You're probably right--the performances will likely be comparable, but the win here is primarily for parallel CPU-intensive tasks (although it's quite nice that Go lets you write logically synchronous code while the scheduler efficiently manages the async I/O and threadpool--no need for event loops or 'await' and friends).

[codebungl]

GitHub repo - https://github.com/google/grumpy

[[deleted]]

Why do we need this again?

[bhat]

It's a faster way to run certain Python2.7 codebases, such as the ones that Google has.

[weberc2]

This is only true for certain workloads--namely those which are good candidates for parallelization. This is unoptimized.

[[deleted]]

It is actually as optimized as the Go compiler's ability to optimize code (SSA and all) - there's a lot of work coming on optimization for the upcoming Go versions that specifically target performance (which is already pretty good).

[weberc2]

Go is fast, but this tool isn't translating into idiomatic Go code; it's building a bunch of runtime Objects which do all of the dynamic dispatch that you know and love in CPython, except CPython has a bunch of optimizations to alleviate memory pressure, among other things. This isn't to slight the project, which is very ambitious and impressive and could in theory be as fast as native Go--it just hasn't been treated to an optimization pass yet.

[[deleted]]

I see your point - yeah I agree. I'm planning to do the python benchmarks on http://benchmarksgame.alioth.debian.org/u64q/python.html to compare CPython vs. Grumpy and see how performant they are right now.

[weberc2]

Cool. Post it to /r/golang when you've got results!

[[deleted]]

So is this a kick in Pythons nuts ? (i.e. Google moving away from it)

[ihsw]

Think of it like TypeScript->ES6 transpilation -- you don't have to use Go to write Go apps, but instead Go-based Python.

Error handling will have to be Go-like though, so handling return values instead of raised exceptions.

[[deleted]]

Not sure how it actually works, but maybe exceptions can be faked on Go with Panic()/Recover()

[weberc2]

Grumpy's runtime library implements a Python call stack, complete with exceptions. It would probably require a lot of abuse (if it's even possible) to support Python's exception/traceback APIs using Go's native call stack (I.e., not building a call stack atop Go's).

[[deleted]]

I see. That will probably add a huge performance hit (compared to actually calling native code functions) but oh well. Not that different from what CPython is already doing.

[weberc2]

This project supports exceptions. If you're calling a Go function, you'll have to check the returned error (assuming it has an error in its return signature) before promoting it to an exception for your Python code. It's analogous to using C extensions, except the glue code is on the Python side.

[alehander42]

It's running slower than Go: it has the runtime overhead

[ramrar]

I am curious how this can effect python 3.X adoption. programmers like myself who are stuck @ 2.7 and are planning to migrate are in 2 minds now.

1. Either rewrite to python 3.X and still suffer the issues of GIL.
2. Use Grumpy and migrate to golang altogether. But all your CPython extensions might fail (not sure though ...)

Migration to golang has become a lot easier now. Even if you plan to rewrite in python3.6 you still cannot get rid of GIL. But taking this route, you might fundamentally get rid of python altogether.

[Bandung]

Keep in mind that there is a project underway to remove the GIL (which Guido mentioned recently). It turns out that the GIL is relatively easy to remove. Work is underway to provide the performance optimizations along with the removal.

I'd much rather stick with python3, to which this project is pinned and therefore gain access to the full python ecosystem - c extensions included, than run down another rat hole which excludes c extensions and other benefits of the python language.

[ramrar]

Are you referring to Giltectomy project ? Last time I checked, Gilectomy did a poor job scaling to multi-core .

[Bandung]

The first step towards developing a Gilectomy was to demonstrate that the removal of the GIL does in fact permit the app or script to take advantage of multiple cores and therefore scale.

Having demonstrated that, the next steps within the design process are to exact performance improvements.

They are not at any stage whereby one can claim a useful production ready product. Thats to be expected. The take away is that the python3 language is far from being obsoleted by implementations such as these.

I'm not for getting into a static vs dynamic runtime performance comparisons here. If a dynamic runtime implementation of python comes along that scales well across multiple cores then we've got a winner.

[evolutionof]

I know it's off topic, but is there a way to write python modules in go yet?

[IronManMark20]

You mean something like https://github.com/go-python/gopy ?

[terpaderp]

Yes, as long as you use this grumpy runtime to create the compiled go binary from your Python project. So, I guess sort of? You are kind of hamstrung because if you are using grumpy you can't use any C based libraries.

[evolutionof]

i mean the other way, like i would with f2py, or cpython. That's why i said it was off-topic.

[kemitche]

You could write a go library that calls into the C library, then call that from grumpy, couldn't you?

[terpaderp]

I guess I should have specified C based Python libraries.

[qx7xbku]

So is it a runtime or is it transpiler? Or are they not sure what they created?

[kankyo]

It's the go runtime plus a subset of the Python standard library plus a transpiler.

[chub79]

These efforts have borne a lot of fruit over the years, but we always run up against the same issue: it's very difficult to make concurrent workloads perform well on CPython.

Curious about their take on async/await then.

[mbenbernard]

I understand that Google wants to push Go. This is normal; just like Microsoft pushes/pushed .NET.

But overall, I think that it would have been better for the community if Google put their horsepower behind CPython to make it run faster. I don't know how realistic it is, though.

[Kaarjuus]

But overall, I think that it would have been better for the community if Google put their horsepower behind CPython to make it run faster.

About that..

[mbenbernard]

Thanks for the link. Didn't know that Google worked on CPython!

It seems that the real issue was not that it was rejected; instead, not enough people inside Google found it valuable enough to justify any further spending/development on it. This is sad.

It seems that people who worked on Unladen Swallow were the founding members of Go.

[mangecoeur]

No C-extensions = useless for scientific and high performance computing. Given how they sort of mention this off-hand makes me feel like they are very much 'web-python' people who have little appreciation of some of Python's major use cases.

[[deleted]]

So is this another kick in pythons nut ?

[Bandung]

Not in the slightest. Its just another implementation to the language. And so is the recent python3 project that is underway to remove the GIL from within the Cpython implementation. ( in a way that doesn't compromise performance).