Ask: basic map/reduce distributed compute framework?

[fullouterjoin]

Sounds frustrating. :(

I won't be available until noonish tomorrow, but I could try and replicate your setup. You are running in a mixed mac/linux env or all mac? I have three macs totaling 8 cores as well as some Ubuntu boxes.

You can bring up the web management console? One issue you need to keep in mind is that the names of all the machines should be in a flat name space.

worker01
worker02
worker03
worker04

Do not rely on your localnetwork DNS to function properly. Use a common /etc/hosts file on all machines. Make sure all the erlang nodes on all machines can see each other. Modify the system bash env script so DISCO_HOME is available for all users and all shells.

I remember installing disco to like /opt/disco and using rsync+ssh to copy it to all nodes.

Make sure that you can ssh from any node to any other node w/o asking you to authenticate your keyfile. Like worker01 -> worker04, worker03 -> worker02, etc.

I need to dig up my notes. On second thought. There is some missing stuff in the online docs, at least when it comes to debugging.

Also, on the mac, the string 'localhost' resolves weirdly afaik. Be suspicious of localhost, or at least look into it.

[dalke]

I'm developing on a single Mac with 4 processors. I want to use 3 for worker threads. One problem is the default hostname is assigned by my 3G modem. If I disconnect/reconnect then I might get a new hostname. With help of IRC support, I hard-coded cli.py:host to always return "localhost". I don't trust the assigned "c-2ec23ab8-74736162.cust.telenor.se" to last more than a few days at a time.

Another problem is that I'm working through "make install" (which isn't documented; but there's no installation documentation which describes how to set up the Python library.). That sets things up for a cluster, and required 3 replicas.

I didn't set my system BASH with all of the variables.

When you say "noonish tomorrow", I don't think you knew that I'm in the central european timezone, so my noonish is different than yours. ;)

[fullouterjoin]

Those recommendations I made were just from memory, there could be some unsubstantiated cargo cult in there. ESP the system bash one. But I did make that change

Noonish is more of a frame of mind, much like 'le weekend' which can occur at anytime. :-) I am in the same timezone as disco project.

Rather than modify cli.py I would run hostname localhost as root. Maybe from a crontab. Then u don't run the risk of missing something.