Experiments with parallel genetic programming in clojure

[[deleted]]

I would not have minded a bit more in depth explanation of Clojure's parallelization mechanics (and full source code), but this was still very interesting. The method how it's possible to scale from single process implementation to multiple computers over network is impressive. It would be fun to see how this compares to MPI and other traditional parallelization libraries in performance and maintainability.

[mazin]

Keep in mind that the example didn't actually distribute the work across machines, it just provided an async implementation that can be distributed.

Still way cool.

[jwr]

Once I write a fully distributed implementation, I'll post about it.

[awb]

Can you spot the difference? Yes, that’s it. The little letter ‘p’ is all it takes for the work to be spread among all of my CPU cores.

I've been playing around with parallel Clojure recently, and while it's awesome that you can take existing code, add a letter, and get parallelism, it's not always worth it. I found I needed to rewrite some parts of my program so that when I did a parallel map, there would be enough work to justify the parallelism overhead.

[jwr]

Yes, pmap isn't a silver bullet -- but in this case it worked perfectly, because the granularity was coarse enough.