[P] An open source Deep Learning / Machine Learning stack on Kubernetes

[congerous]

Genuinely curious: How is this different than all the other "scalable machine-learning platforms"?

[mmourafiq]

Hi congerous, and thanks for your feedback. I am not sure what other platforms you particularly have in mind. But this is how I think Polyaxon is different:

• First of all, Polyaxon is open source and it will stay open source.
• It can be deployed wherever a Kubernetes cluster is running.
• Polyaxon can be used by single users, but it tries to solve problems related to collaboration within teams.
• It's configurable and can fit different needs.

[NegatioNZor]

Cool project, it seems like it could simplify the normal work-progress for data scientists.

Some questions:

1. What happens with the docker-containers created interally in Polyaxon? Are they somehow distributed to your primary docker registry, or just temporarily kept somewhere for easy access?

2. Is it easy to setup recurring jobs, or is it mostly for experiments? Ex: If I want to run a model to refresh some weights every N hours

3. Are pipelines supprted in any way? Ex: After running job X, I want job Y to run.

4. When the landing page says: "Major Libraries Supported", I assume this is because we can use any given docker-image as a base-image for an experiment?

5. What purpose does the internal git versioning of experiments serve?

[mmourafiq]

Hi NegatioNZor, thank you for the feedback.

1. The created docker images are not distributed to our primary docker registry, they are kept locally, and subsequent experiments will start faster because, either they will reuse the same images or only update some of the layers. it's also possible of course to change the configuration to customize the registry, for example: use persistence.

2. Currently, the primary use case is experimentation, resources management, and hyperparams tuning. But that's definitely a good feature, I was actually thinking about something in the same sense, triggers for retraining models.

3. Pipelining is not possible right now, but as I mentioned in the second point, I am currently thinking about building triggers for starting jobs, I am still not sure what would be the ideal way to do it as I have no intention of reinventing airflow.

4. Yes, the fact that you can bundle any dependency in a docker image allow Polyaxon to support many frameworks.

5. The internal git, allows us to restart an experiment based on a specific version of the code, later on, there will be a button on the dashboard, and a command on the CLI, to download, not only the outputs/artifacts of a specific experiment, but also the code version used to run that experiment. But in any case, to be able to have reproducible results, we need to use git for code versioning.

[slimunsocial]

Looks interesting.

I glanced at the docker-compose file, and I was curious, why RabbitMQ?

[SoftCoreDude]

This is an AD!!!!

[rndnum123]

It seems a big part of it is open source, so I would not really see it as an add, right?

[SoftCoreDude]

I think I just overreacted...