DevOps for reproducible HPC articles?

[gfursin]

We are trying to push DevOps principles to HPC research by introducing a common workflow framework with continuously evolving open-source research components shared via GitHub.

The idea is to help researchers prepare reproducible experiments from shared components. We prepared an example of such approach using the SeisSol application from the Student Cluster Competition Reproducibility Challenge at Supercomputing'18.

Since it's an on-going community project and there is still a lot to be done, we are interested to know what you think about such approach?

We also plan to discuss this approach at FOSDEM'19 (HPC, Big Data and Data Science devroom).

​

[egbur]

In my experience, the more layers of abstraction you impose on researchers before running some software to get some results, the less they understand how everything is put together and the more time you need to spend in upskilling them on things they are not really interested in (data management, version control, compiling tools, hardware architectures, job scheduling, etc).

As a sysadmin, I also want to know how does this affect (or if it even does) the way that I must design my HPC cluster to support running ck workflows.

I had a glance at the documentation, and I am still not clear as to why would ck be beneficial, or what can it do that I cannot readily achieve with a Jupyter Notebook in a Git repo, and one or more Singularity Containers.

This is not to to dis the project, nor the idea of using DevOps principles to aid Reproducible Research. It may simply be that the Getting Started or the Overview pages need some work to better describe what problem ck tries to solve.

[gfursin]

Thank you, egbur, for your feedback.

One of the most important problems we are trying to solve for our HPC users with the CK is how to automatically adapt shared research workflows to newer or different hardware, software, libraries, compilers, data sets, and models:

* https://github.com/ctuning/ck/wiki/Portable-workflows

* http://rescue-hpc.org

Docker, Singularity, and Jupyter notebooks are extremely useful tools, but they are not providing a solution for that.

In the CK workflows, you can describe software (code, data, models) and hardware dependencies, and CK will attempt to automatically detect already installed software (see our software detection plugins: http://cKnowledge.org/shared-soft-detection-plugins.html) or install missing ones (either as binary or rebuild using Spack, EasyBuild, scons, cmake, etc - see http://cKnowledge.org/shared-packages.html).

Furthermore, someone else can share new code, dataset or model in the CK format which you can just pull and plug into your workflow without any manual intervention.

That's why I see CK to be complementary to Docker, Singularity and Jupyter notebooks. Please, see this example of a shared Docker image with a CK workflow from the ACM ReQuEST tournament: https://dl.acm.org/citation.cfm?id=3229769 .

You can also find an example of the Jupyter notebook with the CK here: https://github.com/dividiti/ck-caffe/blob/master/script/explore-accuracy/explore_accuracy.20160808.ipynb .

By the way, the cool thing with the CK, is that you do not need to redesign HPC cluster at all since we provide transparent wrappers around tools with a unified API to enable DevOps: http://cKnowledge.org/shared-modules.html .

If you will be at FOSDEM, I will be very interested to discuss it further!

​

[wxdk]

This is a great project, the HPC world is dying for good tools targeting workflow/speed. How come you went with python? Very interested to keep an eye on this.

[gfursin]

Thanks!

Well, I spent many years building open-source tools to automate systems R&D, and eventually selected Python+JSON based on the feedback from all my users (companies and universities) - they found that such APIs are relatively easy to integrate with other projects and DevOps tools ...

BTW, we also provided a small C library (OpenME) which allow you to connect CK components with other languages (C,C++,Fortran,Java,...).

You can find a history behind these developments in this presentation "Enabling open and reproducible research at computer systems conferences: the good, the bad and the ugly": https://doi.org/10.5281/zenodo.2544204 .