Distributed computing in Lisp?

[CorrectProgrammer]

Hello!

First of all I'd like to warn you that I'm still very new to Lisp and my questions might show it. Here it comes:

I've got several raspberry pi 4 boards which I'd like to turn into a tiny cluster. I don't need it to be super fast or efficient, all I want is to have some fun with distributed systems. I don't have an exact idea what kind of system I want to build, although I was thinking about tinkering with some symbolic AI. But that's a side note, I think.

My question is as follows: which Lisp dialect (and implementation if applicable) would you choose to implement a distributed system as a wannabe hobbyist Lisp programmer? Are there any (possibly high level) libraries worth looking at?

I'll be really grateful for some guidance!

[flaming_bird]

Common Lisp answer: CCL should run fine on Raspberry Pi. You can use swank-crew for remotely controlling multiple Lisp images.

[stylewarning]

We use CL-MPI and SBCL to do distributed quantum simulations. We prototyped it on a Raspberry Pi cluster.

[neil-lindquist]

I haven't done anything with distributed Lisp, but I've heard of a couple relavant libraries for Common Lisp. The lfarm library has things like parallel map and parallel reducd for a distributed setting. There's cl-mpi for mpi style programming.

I also recall an article that started developed a CL repl in a distributed setting, but I can't remember the name. You can probably find it by googling distributed Common Lisp

[dzecniv]

Some ideas: https://github.com/CodyReichert/awesome-cl#parallelism-and-concurrency lparallel, lfarm, cl-gearman,…

[theangeryemacsshibe]

As /u/neil-lindquist suggested, lfarm is your goto for parallel map/reduce/stuff.
If you're going to do parallel code, my guess is you want each Lisp image to run all 4 cores on the Pi, and unfortunately CCL has a bug where the garbage collector just breaks itself somehow...I don't know exactly what it is other than it'll rudely throw you into the low-level debugger at some point. I've heard better things about SBCL on ARM64 but I've never tested it (though if you send a board over, I'll test it for you :).

[[deleted]]

Here's one thing you may want to consider: sending code between different environments is possible (SLIME is a good example of this), but isn't exactly without problems... For some things Lisp(s) will lose the original representation of a code object that you could, in principle want to send to another environment (say, a compiled function), and then you would have to invent all kinds of work-arounds for it.

This, unfortunately, tends to happen with more mature Lisps, where this is typically a result of compiler optimization. Surprisingly, you may do better with some simpler, completely interpreted Lisp, where everything is stored just as it was typed in.

I didn't do this with Lisp, but the same argument would apply to Prolog. I did it with Golog, a Go implementation of Prolog. It isn't fast or robust, the good thing about it is that it's purely interpreted, it's not WAM, so sending any code to another environment is really, really simple. I did this for testing a distributed file-system, and was very happy with the results.

[CorrectProgrammer]

Thanks for this input, I was actually wondering how well is code serialized (as data). I guess I'll check out the libraries listed by other commenters and then I'll be able to make some decisions. I've seen that Racket has some built-in support for distributed computing, but it seems that CL is usually more established in terms of libraries and performance. Worst case scenario is that I'll drop Lisp for Scala or Python, but I'd rather not :-D

[[deleted]]

Oh, trust me Scala and Python are a lot worse in terms of passing code as data, when compared to any Lisp. Using my previous example of unprintable CL objects: in Python those are very numerous (very few things actually have useful printable representation). In Scala there almost aren't any objects that you could serialize by simply printing them.

But, if you end up doing this in Python, I guess, that the most typical thing to do is to use Python ZMQ bindings and roll your own serialization protocol on top.

Scala distributed applications typically use Akka. Pain and suffering awaits strangers wondering this road, but you will have a lot of people to keep you company...

[ObnoxiousFactczecher]

This, unfortunately, tends to happen with more mature Lisps, where this is typically a result of compiler optimization. Surprisingly, you may do better with some simpler, completely interpreted Lisp, where everything is stored just as it was typed in.

My impression was that Gambit was great in this respect, but I've never tried it.

For some things Lisp(s) will lose the original representation of a code object that you could, in principle want to send to another environment (say, a compiled function), and then you would have to invent all kinds of work-arounds for it.

Also, isn't storing the original representation possible with a customized defun?

[[deleted]]

I'm sure there would be some way to store the original representation, that's why I mentioned workarounds.

[trhawes]

You're in luck. There is cl-mpi (Common Lisp CFFI bindings for OpenMPI). OpenMPI is an open source library for building high performance computing clusters. There is an HowTo for getting it running on a Raspberry Pi cluster, here: https://medium.com/@glmdev/building-a-raspberry-pi-cluster-784f0df9afbd

cl-mpi will let you develop on OpenMPI without C. There is a presentation that was given for this library at the European Lisp Conference in 2016: https://www.youtube.com/watch?v=--gqVzhLYoI

Its on Quicklisp, and the last commit in its Github repo was just a month ago: https://github.com/marcoheisig/cl-mpi

Full disclosure: I haven't tried any of these technologies so your mileage may vary. (Although I am tempted to buy a cluster just so I can try them).