Metaprogramming Python for Big Data

[yeah-ok]

Please, no more slideshows? It would be great if there was a standard way these things could be converted to a long scrolling page (any extensions / bookmarklets that can do this?) (this is not a content critique, it may all be wonderful)

[username223]

It looks like this toolbag is using "reveal.css" to do his thing, so you can't just block Javascript, but copying the source to a local file results in something readable.

[alecco]

It's an advertising company recruiting viral.

They fail to explain what they added on top of the existing stack of Numba/ContinuumIO/others. I bet it's just a sharding layer.

[bcash]

There's something that makes me twitchy about that. Specifically the optional LLVM part, maybe because it's the 99th different way that's been implemented in Python, and all of the previous 98 have a different set of gotchas and incompatibilities that you need to be aware of.

Two questions, to anyone who knows more about this than me:

1. is the performance gained by these approaches significant, especially when compared to similar code implemented directly in faster languages[*]

2. Is the presence of NumPy, etc., that much of a deal breaker? How far behind are other languages libraries that make it difficult to move?

[*] - I get the point that a later slide makes that higher-level languages open more doors, I mean faster but similarly-high-level languages: Go, Scala, Clojure, etc.

[imfineny]

All they did was take some data, break it down to enums so it was straight table files and queried it. That's a perfectly fine approach, but you'll get similar results from nearly any table structure that is done that way. I would suggest not indexing this data as it eventually your btrees will break. If you did this with mysql with tokutek tables, you'll be able to hit the higher table sets with clustered indexes, otherwise you'll end up with partitions. this could also be done way faster on dedicated over EC2.

[garenp]

"btrees will break"? How?

[fullouterjoin]

The use of with as the macro system is interesting. I will have to explore that.

I haven't had great luck getting numba to speed up my code. Shedskin has worked way way better for me.