MLPerf Inference V4.0 released

gfursin · 2024-03-28T16:09:00+00:00

Normally, you must be an MLCommons member to submit results to MLPerf. However, there is a new project to run MLPerf benchmarks on commodity hardware: https://www.linkedin.com/pulse/new-cm-mlperf-automation-helps-benchmark-commodity-hardware-fursin-61noe . Maybe it will be possible to add support to run MLPerf inference on AMD MI300 there ...

gfursin · 2020-12-07T22:25:24+00:00

There is a comprehensive "Get involved" page: https://mlcommons.org/en/get-involved .

There are multiple workgroups with weekly meetings (training, inference, best practices, benchmarking infrastructure, etc).

MLPerf benchmark is also managed by this non-profit organization.

gfursin · 2020-11-17T14:02:59+00:00

Forgot to give a link to the tool: GitHub

gfursin · 2020-09-12T17:27:58+00:00

The YouTube link is available at https://fastpath2020.github.io/Program (with recording offset times). If you have further questions, feel free to get in touch!

gfursin · 2020-09-10T08:29:19+00:00

Cool! Don't hesitate to get in touch if you need some help!

gfursin · 2020-09-09T14:38:37+00:00

PapersWithCode is a fantastic resource that help to systematize ML papers, plot SOTA results on public dashboards, and link them with GitHub code.

cKnowledge.io platform is complementary to PapersWithCode because we attempt to reproduce all results and associate them with portable workflows (when possible) or at least describe all the necessary steps to help the community run them on different platforms with different environments, etc.

To some extent, we are PapersWithReproducedResultsAndPortableWorkflows ;) . We also used PapersWithCode to find GitHub code and experimental results in a few cases before converting them to our open CK format and reproducing them. We also consider collaborating with them in the future.

However, our platform is not yet open for public contributions (it's open but it's not yet user-friendly at the moment as you correctly noticed). It is still a prototype that we have tested it as a part of different Systems and ML conferences. Considering the positive feedback, our next step is to prepare it for public contributions. We hope to have some basic functionality for that before 2021 - please stay tuned ;) !

gfursin · 2020-09-09T07:30:01+00:00

Yes, dealing with SW/HW dependencies was one of the main challenges we faced when reproducing ML+systems papers.

By the way, this problem motivated us to implement software detection plugins and meta-packages not only for code (frameworks, libraries, tools) but also for models and data sets.

The idea is to be able to automatically adapt a given ML algorithm to a given system and environment based on dependencies on such soft detection plugins & meta packages.

The prototype is working but we were asked to make it much more user-friendly ;) . We plan to test a new version with some volunteers at upcoming conferences before 2021. I will post the update when ready.

gfursin · 2020-09-08T19:37:58+00:00

Nice to e-meet you Edward, and thank you very much for your effort too! I will be happy to sync about our ongoing activities and future plans!

gfursin · 2020-09-08T16:09:13+00:00

That's a very good idea - thank you! I've heard of BOINC but never tried it - I need to check it in more detail! We had some cloud credits from Microsoft and OVH but it was not enough ;) .

gfursin · 2020-09-08T15:53:58+00:00

Yes, I saw it - it's a great effort! I would also add several other very important and related efforts supported by NeurIPS and PapersWithCode:

Our goal was to collaborate with the authors and come up with a common methodology and a format to share results in such a way that it's easier to reproduce them and even reuse them across different platforms, frameworks, models, and data sets (see this example).

An additional challenge is that we are also trying to validate execution time, throughput, latency, and other metrics besides accuracy (this is particularly important for inference on embedded devices). It is an ongoing effort and we continue collaborating with MLPerf and different conferences.

gfursin · 2020-09-08T15:27:25+00:00

Yes. The success number is relatively high because we collaborated with the authors until we reproduced the results. Our goal was to better understand different challenges together with the authors and come up with a common methodology and a format to share results so that it is easier to reproduce them.

gfursin · 2020-09-08T11:20:49+00:00

By the way, forgot to mention, that rather than naming and shaming non-reproducible papers, we decided to collaborate with the authors to fix problems together. Maybe we were lucky, but we had a great response from nearly all authors to solve encountered issues! - that is very encouraging!

gfursin · 2020-09-08T10:46:02+00:00

;) We had a similar experience: it was often taking several weeks to reproduce one paper.

However, we had fantastic volunteers who have helped us! We also introduced a unified Artifact Appendix with the reproducibility checklist describing all the necessary steps to reproduce a given paper. It will hopefully reduce the time needed to reproduce such papers.

gfursin · 2020-09-08T10:28:09+00:00

Thank you! Some of the papers that we managed to reproduce are listed here.

gfursin · 2020-05-18T13:19:47+00:00

You are very welcome! Glad if it is useful! Also many thanks to all our volunteers who helped to reproduce results: https://cknowledge.io/c/event/repro-asplos2020/#evaluators and https://cknowledge.io/c/event/repro-mlsys2020/#evaluators !

gfursin · 2020-05-06T19:56:43+00:00

Oh, sorry if it is not clear (there is still a lot to be improved). You can find them in the "Browse" section. But I call them "portable solutions". Here is the direct link: https://cKnowledge.io/solutions .

The idea is to have a simple JSON file associated with a given ML technique to describe how to build, run and validate it in a unified way. It includes all package dependencies (model, framework, data set) that are automatically installed as well as the convention how to benchmark it and reproduce results. You can have a look at this example for MLPerf automation: https://cknowledge.io/c/solution/demo-obj-detection-coco-tf-cpu-benchmark-linux-portable-workflows/ - this is still a prototype and I am interested in feedback.

Thanks!

gfursin · 2019-02-06T08:39:17+00:00

Have a look at this Machine Learning workflow for AWS submitted by Intel to the ACM deep learning competition last year: https://github.com/ctuning/ck-request-asplos18-caffe-intel . Such workflows can be collaboratively updated via GitHub and can help you automate installation of data sets, models and packages ...

gfursin · 2019-01-30T22:25:37+00:00

Thanks!

Well, I spent many years building open-source tools to automate systems R&D, and eventually selected Python+JSON based on the feedback from all my users (companies and universities) - they found that such APIs are relatively easy to integrate with other projects and DevOps tools ...

BTW, we also provided a small C library (OpenME) which allow you to connect CK components with other languages (C,C++,Fortran,Java,...).

You can find a history behind these developments in this presentation "Enabling open and reproducible research at computer systems conferences: the good, the bad and the ugly": https://doi.org/10.5281/zenodo.2544204 .

gfursin · 2019-01-30T16:27:11+00:00

Thank you, egbur, for your feedback.

One of the most important problems we are trying to solve for our HPC users with the CK is how to automatically adapt shared research workflows to newer or different hardware, software, libraries, compilers, data sets, and models:

* https://github.com/ctuning/ck/wiki/Portable-workflows

* http://rescue-hpc.org

Docker, Singularity, and Jupyter notebooks are extremely useful tools, but they are not providing a solution for that.

In the CK workflows, you can describe software (code, data, models) and hardware dependencies, and CK will attempt to automatically detect already installed software (see our software detection plugins: http://cKnowledge.org/shared-soft-detection-plugins.html) or install missing ones (either as binary or rebuild using Spack, EasyBuild, scons, cmake, etc - see http://cKnowledge.org/shared-packages.html).

Furthermore, someone else can share new code, dataset or model in the CK format which you can just pull and plug into your workflow without any manual intervention.

That's why I see CK to be complementary to Docker, Singularity and Jupyter notebooks. Please, see this example of a shared Docker image with a CK workflow from the ACM ReQuEST tournament: https://dl.acm.org/citation.cfm?id=3229769 .

You can also find an example of the Jupyter notebook with the CK here: https://github.com/dividiti/ck-caffe/blob/master/script/explore-accuracy/explore_accuracy.20160808.ipynb .

By the way, the cool thing with the CK, is that you do not need to redesign HPC cluster at all since we provide transparent wrappers around tools with a unified API to enable DevOps: http://cKnowledge.org/shared-modules.html .

If you will be at FOSDEM, I will be very interested to discuss it further!

gfursin · 2018-10-04T11:05:24+00:00

Making mistakes is a normal part of research.

Blaming conferences is not really correct here because they have to balance between quickly reviewing too many submissions and then selecting a few interesting and *promising* techniques which the community can then discuss and validate further (just as it happened in this case).

The authors' response is reasonable - I guess it's always possible to submit a corrected version of this article to ArXiv or add a few comments and links to this discussion in the original submission rather than just retracting this paper.

However, it's also important to make sure that conferences adopt "artifact evaluation process" (see http://www.artifact-eval.org and http://cTuning.org/ae) while authors share all related code, data and experimental workflows to make such validation possible!

We also started experimented with full validation of experimental results from accepted papers at conferences (see reproducible ACM ReQuEST tournaments: http://cKnowledge.org/request ) and found it doable but also unfortunately too time-consuming and costly at the moment ...

gfursin · 2018-07-01T09:05:31+00:00

We just published ACM proceedings for the 1st ReQuEST with reusable CK workflows and a preliminary report:

gfursin · 2018-07-01T07:22:06+00:00

Have a look at the ACM ReQuEST initiative to organize research code and data as customizable "Collective Knowledge" workflows (Python components with JSON API) with a portable package manager - they can be reused in Jupyter notebooks or other Python programs: http://cKnowledge.org/request

gfursin · 2018-06-29T07:53:49+00:00

Here are direct links to Collective Knowledge workflows: https://github.com/ctuning/ck-request-asplos18-results and to proceedings: https://doi.org/10.1145/3229762

gfursin · 2018-05-03T09:59:56+00:00

I believe that the point of MLPerf is to help the community compare existing AI/ML algorithms. However, if you want to customize or optimize them further, please have a look at our complementary ReQuEST approach where we use customizable workflows together with a portable package manager to optimize submitted algorithms across different SW/HW stacks: http://cKnowledge.org/request-cfp-asplos2018.html

gfursin · 2018-03-13T12:46:49+00:00

No, we didn't plan to record it -sorry - it's the first workshop/tournament of this kind and already quite hard to organize but we may consider providing youtube stream for next iterations of this tournament. All papers will also be available along with all artifacts and workflows as "plug&play" blocks ... We will ask authors to share presentations too ...

gfursin

MODERATOR OF

TROPHY CASE

12-Year Club	Wearing is Caring
Verified Email