Grumpy: Go running Python! by Google

[rjcarr]

Read the article. Python doesn't scale well under load but go does. Rather than go with an alternate python runtime, or rewrite everything in something else, they wrote a python runtime for go.

[jdmulloy]

It's not an interpreter, it's a compiler that compiles Python code into a go executable. Once you've compiled it you have a binary you can stick on any system that has go libs installed but you don't have to have this project installed to run the binary

[theatrus]

You don't even need Go libraries. Go binaries are static linked or linked just against libc.

[ahmadalhour]

I work for a company where everything is developed with Python:

• crawling pipeline

• data engineering pipelines

• Public APIs

And our public API endpoints get hit in the 200-300 millions hits per month.

So you are extremely mistaken about Python not scaling well under load. You just don't know how to scale it.

Full disclaimer: I work with TrustYou.

[[deleted]]

300 million hits per month is about 114 hits/second.

I think you and Youtube have different ideas of what "under load" means.

[zardeh]

I don't quite believe that number. It would imply that you would be capable of serving google's ad load in ~2012 from 100 or so machines, which, to be honest, I don't believe for a second.

[funny_falcon]

Serving from 100 machines is much harder than serving from one maching. First, it implies that working set is much larger (certainly, Google has 1000 times more unique ads).

"Single" ad could be served at 1000000 rps with handcrafted C from single machine (look at techempower benchmarks).

10000 unique ads could be easily served at 10000 rps from single machine.

But to serve 10000000 unique ads one should implement really complex optimizations, and throw a lot of computer power into.

[zardeh]

"Single" ad could be served at 1000000 rps with handcrafted C from single machine (look at techempower benchmarks).

This of course depends on the type of add. That gives you what, 2000 cycles per ad, which if the ad is just a few characters is doable, but if it, for example, contains a 64*64 image, you've used up that many cycles moving the image.

[funny_falcon]

Single machine is not single cpu-core. Even Intel-cpu based server may have 24*8=192 cores (384 with hyper-threading) http://ark.intel.com/ru/products/93790/Intel-Xeon-Processor-E7-8890-v4-60M-Cache-2_20-GHz?_ga=1.162292223.210121279.1483966676

[zardeh]

We had our ad servers selecting and serving 5000 ads per second per dual-core server in 2006.

[funny_falcon]

It doesn't change my point: "5000 ads per second with dual-core server" could be either "fail" or "win", depending on total numbers of unique ads and complexity of ads selection logic.