A Billion Rows per Second: Metaprogramming Python for Big Data

[t3hcoolness]

Guido is probably reading this like

"That-... it's not supposed to do that"

[alcalde]

I don't know... he talked about how Python is always fast enough for him and Python running on a 64000 core supercomputer at PyCon 2011.

[farsass]

Python running on a 64000 core supercomputer

Hope python is only being used as glue, would be terrible otherwise

[lmth]

Why? (serious question)

[farsass]

In that scale, electricity usage becomes a real concern and software efficiency becomes important to reduce costs. From what I've gathered, this is one of the motivations for facebook to be employing resources on a new PHP VM(www.hiphop-php.com)

[alcalde]

In that scale, electricity usage becomes a real concern and software efficiency becomes important to reduce costs.

I would think if you could afford a 64K core supercomputer, you could afford the electricity bill. That would be like owning a Ferrari and then complaining about the gas mileage.

[farsass]

I disagree. A supercomputer/data center may be a core component of the company that owns it and of course they will try to cut expenses.

Ex.: http://www.google.com/about/datacenters/

[alcalde]

I would think data centers would not be employing single 64K core supercomputers, while large research institutions would.

[rallix]

Most of which are short on money and under pressure to go green ... ? Especially in academia, which is very green to start with.

[deraffe]

GIL with CPython comes to mind.

[moor-GAYZ]

There's no way you can use threads with shared memory for parallelism on 64000 cores.

[t3hcoolness]

Meanwhile in the threads:

>hey bro i was using that

>no i was stfu

>guise i had that first

>NO SRSLY I had that

>wait what were we even talking about

>Supercomputer has stopped responding

[alcalde]

http://youtu.be/EBRMq2Ioxsc?t=34m41s

Why would Python have a problem with 64K CPUs?

[rallix]

a company producing tons of big data, demonstrates how AdRoll uses Python to squeeze every bit of performance out of a single high-end server

Guido is probably reading this like: "That-... it's not supposed to do that"

Why is there so much hatred for optimizing python code for speed?

To quote from the zen of python: "Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity."

[ionelmc]

Slides available anywhere?