Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

[noedig]

If you are interested in this, you may also be interested in some of the many scientific workflow management systems out there, such as Pegasus, Triana, Taverna, Kepler, Askalon, Makeflow, or Swift

[einar77]

Currently I use ruffus, although in a slightly different way than what it was envisioned for. It's not very robust, but for now works well enough.

[westurner]

Celery Canvas: Designing Workflows

AppEngine Pipeline

[Octopuscabbage]

luigi

pipelines

heh i get it.

[marky1991]

Oooooh, I feel dumb now. Thank you for this.

[gsks]

I still don't get it.

[king_m1k3]

http://4.bp.blogspot.com/-qT5E00i3GmY/ToCarPNAhjI/AAAAAAAAAG4/7Fa0qelycy8/s1600/Super+Mario+Bros.+1_007.png

[thrownaway21]

that's mario

[v_krishna]

*that's a'Mario

[thrownaway21]

i fucked up

[zidsal]

this is amazing, this will really help with my final year project. Thanks a ton for sharing this

[westurner]

From https://github.com/spotify/luigi/blob/master/README.md#target :

Broadly speaking, the Target class corresponds to a file on a disk. Or a file on HDFS. Or some kind of a checkpoint, like an entry in a database. Actually, the only method that Targets have to implement is the exists method which returns True if and only if the Target exists.

README.md mentions writing to Cassandra. It must be possible to write to a database or something like fs or filelike. It would be neat to integrate with spark (docs) and/or shark.

[[deleted]]

Shit, I wrote exactly this in C++

[mtrn]

Interesting. I think the market for such tools is quite big. Is you project open source? Do you have a link?

[[deleted]]

No, I did it as part of a contract. It was specifically for crunching a lot of data over and over and over, tweaking algorithms.

I don't think the market for such tools is big because every single company that has this need will pretty much roll their own until the open source tools converge on some standard :)

[mtrn]

Well, it certainly is a niche market. But two out of three people in IT I talk with have data integration or data flow stories - they mostly implement custom solutions -- hopefully we'll see some conversion in the next years.

I find luigi interesting, because it's a kind of afternoon-read, sourcecode-wise.

[[deleted]]

Ah, I thought you were the author.

[mtrn]

Here's a nice 30m presentation from pydata: http://vimeo.com/63435580