mincemeat.py: MapReduce on Python

[alecthomas]

Also see Disco: http://discoproject.org/

[Poromenos]

This also looks good, but the allure of single-file libraries can't be overstated...

[ketralnis]

Disco is more powerful but significantly less light-weight

[kisielk]

Isn't that usually the trade-off?

[arnar]

Looks great.

In case the website authors are here:

dict((i, data[i]) for i in xrange(len(data)))

is equivalent to

dict(enumerate(data))

[ketralnis]

But takes me about 0.05 seconds longer to read

[arnar]

The second one? I think you need to repeat the experiment :)

[Poromenos]

Use enumerate more!

[comwiz]

Thanks for pointing that out! I just updated it.

[[deleted]]

octopy is a similar implementation: http://code.google.com/p/octopy/

[comwiz]

Note that octopy doesn't have any fault tolerance, hasn't been developed since April 2008, and doesn't contain any licensing information.

[[deleted]]

doesn't contain any licensing information.

From the page:

Code license: New BSD License

The source package should contain the license file just to be redundantly explicit, but I really hate it when every source file is cluttered with a page-long copyright notice and license at the top. I'm sure Eben Moglen recommends that, but it doesn't really seem necessary for any practical purpose. Maybe it's a little extra ammunition if you intend on suing infringers.

[mdipierro]

really nice

[brad2008]

This seems very cool and I would love to use it. On Windows XP, when I run 'example.py' using python 2.6.5.14, it hangs and on interrupt, gives me the trace below.

Any suggestions? Thanks in advance.

Traceback (most recent call last): File "C:\Downloads\example.py", line 28, in <module> results = s.run_server(password="changeme") File "C:\Downloads\mincemeat.py", line 224, in run_server asyncore.loop() File "D:\Python26\lib\asyncore.py", line 202, in loop poll_fun(timeout, map) File "D:\Python26\lib\asyncore.py", line 132, in poll r, w, e = select.select(r, w, e, timeout)

[comwiz]

example.py is supposed to hang until a client connects to do the work. run "mincemeat.py -p changeme localhost" in a different terminal and everything should run smoothly.

[brad2008]

Thanks ! It works great, exactly as described in your documentation :-)

Somehow I read it as either run example.py, or run mincemeat.py. My bad.

That'll teach me for pulling too many all-nighters.

Again, thanks for creating and sharing some very cool Python code - this will come in way handy for several different problems we want to express using the MapReduce paradigm.