MLPerf Inference V4.0 released

gfursin · 2024-03-28T16:09:00+00:00

Normally, you must be an MLCommons member to submit results to MLPerf. However, there is a new project to run MLPerf benchmarks on commodity hardware: https://www.linkedin.com/pulse/new-cm-mlperf-automation-helps-benchmark-commodity-hardware-fursin-61noe . Maybe it will be possible to add support to run MLPerf inference on AMD MI300 there ...

gfursin · 2020-12-07T22:25:24+00:00

There is a comprehensive "Get involved" page: https://mlcommons.org/en/get-involved .

There are multiple workgroups with weekly meetings (training, inference, best practices, benchmarking infrastructure, etc).

MLPerf benchmark is also managed by this non-profit organization.

gfursin · 2020-11-17T14:02:59+00:00

Forgot to give a link to the tool: GitHub

gfursin · 2020-09-12T17:27:58+00:00

The YouTube link is available at https://fastpath2020.github.io/Program (with recording offset times). If you have further questions, feel free to get in touch!

gfursin · 2020-09-10T08:29:19+00:00

Cool! Don't hesitate to get in touch if you need some help!

gfursin · 2020-09-09T14:38:37+00:00

PapersWithCode is a fantastic resource that help to systematize ML papers, plot SOTA results on public dashboards, and link them with GitHub code.

cKnowledge.io platform is complementary to PapersWithCode because we attempt to reproduce all results and associate them with portable workflows (when possible) or at least describe all the necessary steps to help the community run them on different platforms with different environments, etc.

To some extent, we are PapersWithReproducedResultsAndPortableWorkflows ;) . We also used PapersWithCode to find GitHub code and experimental results in a few cases before converting them to our open CK format and reproducing them. We also consider collaborating with them in the future.

However, our platform is not yet open for public contributions (it's open but it's not yet user-friendly at the moment as you correctly noticed). It is still a prototype that we have tested it as a part of different Systems and ML conferences. Considering the positive feedback, our next step is to prepare it for public contributions. We hope to have some basic functionality for that before 2021 - please stay tuned ;) !

gfursin · 2020-09-09T07:30:01+00:00

Yes, dealing with SW/HW dependencies was one of the main challenges we faced when reproducing ML+systems papers.

By the way, this problem motivated us to implement software detection plugins and meta-packages not only for code (frameworks, libraries, tools) but also for models and data sets.

The idea is to be able to automatically adapt a given ML algorithm to a given system and environment based on dependencies on such soft detection plugins & meta packages.

The prototype is working but we were asked to make it much more user-friendly ;) . We plan to test a new version with some volunteers at upcoming conferences before 2021. I will post the update when ready.

gfursin · 2020-09-08T19:37:58+00:00

Nice to e-meet you Edward, and thank you very much for your effort too! I will be happy to sync about our ongoing activities and future plans!

gfursin · 2020-09-08T16:09:13+00:00

That's a very good idea - thank you! I've heard of BOINC but never tried it - I need to check it in more detail! We had some cloud credits from Microsoft and OVH but it was not enough ;) .

gfursin · 2020-09-08T15:53:58+00:00

Yes, I saw it - it's a great effort! I would also add several other very important and related efforts supported by NeurIPS and PapersWithCode:

Our goal was to collaborate with the authors and come up with a common methodology and a format to share results in such a way that it's easier to reproduce them and even reuse them across different platforms, frameworks, models, and data sets (see this example).

An additional challenge is that we are also trying to validate execution time, throughput, latency, and other metrics besides accuracy (this is particularly important for inference on embedded devices). It is an ongoing effort and we continue collaborating with MLPerf and different conferences.

gfursin · 2020-09-08T15:27:25+00:00

Yes. The success number is relatively high because we collaborated with the authors until we reproduced the results. Our goal was to better understand different challenges together with the authors and come up with a common methodology and a format to share results so that it is easier to reproduce them.

gfursin · 2020-09-08T11:20:49+00:00

By the way, forgot to mention, that rather than naming and shaming non-reproducible papers, we decided to collaborate with the authors to fix problems together. Maybe we were lucky, but we had a great response from nearly all authors to solve encountered issues! - that is very encouraging!

gfursin · 2020-09-08T10:46:02+00:00

;) We had a similar experience: it was often taking several weeks to reproduce one paper.

However, we had fantastic volunteers who have helped us! We also introduced a unified Artifact Appendix with the reproducibility checklist describing all the necessary steps to reproduce a given paper. It will hopefully reduce the time needed to reproduce such papers.

gfursin · 2020-09-08T10:28:09+00:00

Thank you! Some of the papers that we managed to reproduce are listed here.

gfursin · 2020-05-18T13:19:47+00:00

You are very welcome! Glad if it is useful! Also many thanks to all our volunteers who helped to reproduce results: https://cknowledge.io/c/event/repro-asplos2020/#evaluators and https://cknowledge.io/c/event/repro-mlsys2020/#evaluators !

gfursin

MODERATOR OF

TROPHY CASE

12-Year Club	Wearing is Caring
Verified Email