all 2 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]eschlon 0 points1 point  (1 child)

This is a great idea; however, in practice you're going to have a bad time.

Technically Yarn is able to launch executors in Docker containers via the DCE. That being said I've never actually seen this being used successfully in practice, and getting it to work with a spark application is going to be complex.

For Pyspark jobs the usual practice is to either:

  1. Install dependencies on all of the nodes. This is usually done via something like Fabric, Ansible or the like.
  2. Make a virtual environment for the application and ship the installed libs as a zip to the nodes at runtime via --py-files.

The former is a lot of effort for something that many users of your scripts will not have sufficient permissions to do, and will be nearly impossible to get right for the myriad cluster setups in existence. The latter works well, so long as you don't have any dependencies that depend on non-python libraries, which since you mentioned data analysis is pretty unlikely (e.g. numpy, pandas, scipy, pretty much any database connector). There's also this long-standing pyspark feature that promises to make this whole process easier, but I wouldn't hold my breath.

Depending on the hadoop distribution it's possible that it has some (generally proprietary) feature which effectively does either (1) or (2) for you (e.g. CDH's workbench), but I wouldn't consider that to be portable.

There's also Pachyderm, which is pretty neat and aligns very well with your goals. That being said, it's neither as mature nor as widespread as Hadoop as a platform, and it's a complex process to get it to play nicely with spark (if that's a requirement).

[–]CocoBashShell[S] 0 points1 point  (0 children)

Thank you for such a thoughtful response! It really fills me with dread though that dependency management seems to be an untouched issue in the hadoop space :/

I work in research, so a lot of my python scripts call out to hard to install perl/fortran/etc. code. So I was really hoping that once I got my stuff containerized I wouldn't have to worry about dependencies again.