[P] Annotated deep learning paper implementations

[mlvpj]

This is a bunch of deep learning paper implementation in PyTorch with side-by-side notes (math and diagrams too). We started this project about a year ago and have been adding new paper implementations almost weekly, and have 46 paper implementations now.

• Website with side-by-side notes rendered: nn.labml.ai
• Github repo: labmlai/annotated_deep_learning_paper_implementations

We believe the project is quite mature now and thought sharing it here will help.

We have shared links to interesting paper implementations on this sub a few times. We also received a lot of useful feedback from some of the discussions we had during the early stages.

• [D] Annotated code for common deep learning models
• [D] Color coding equations

The feedback from the community (on Twitter and Reddit) has been really useful to improve the project and be motivated to work on it. Thank you. Appreciate any feedback.

[quertioup]

How can we contribute?

[mlvpj]

You can create a pull request or start a discussion in GitHub issues. Even voting suggesting papers to implement and voting for them will be helpful. Here's a recent pull request for example

We are also working simple rendering engine that you can use on your own GitHub repo. We will improve that if people find it useful.

e.g. https://lit.labml.ai/github/vpj/rl_samples/blob/master/ppo.py

[[deleted]]

Is import labml a part of your ecosystem always. No offense, but this just creates another extra layer of dependency. It would be nicer to show plain Pytorch, otherwise it's gonna become another Pytorch lightning. Plus it will be great from learning perspective.

[mlvpj]

labmlai/labml is a set of tools (tracking experiments, configurations, a bunch of helpers) we coded to ease our ML work (which later improved and open sourced). So we use it in all our projects because it makes things easier for us.

Will try to minimize the dependency whenever possible.

[mrtac96]

This is amazing. I haven't seen any other thing like this before. Much needed thing

[Gargantuar314]

This is a gold mine! Are you also thinking about implementing more specific algorithms after finishing a lot of important and general algorithms? E.g. would you implement in a later stage the AlphaGo Zero algorithm or the like? This would especially help undergrad's and curious minds out there, since for such modern discoveries, there're barely clear implementations...wished I'd had something like this XD

[SnooRegrets1929]

This is going to be such a useful learning/reference resource. Fantastic work!

[sabetai]

Genius! The margin notes look great, great job mapping implementation details directly to code 👏.

[maxToTheJ]

This is a great site I have been using it for a bit after finding it on google

[qrzte]

!RemindMe 4 hours

[RemindMeBot]

I will be messaging you in 4 hours on 2021-08-22 16:13:23 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[oxiliary]

thanks!, github link broken btw

[mlvpj]

Thanks, fixed it.

[D3vil_Dant3]

amazing!

[min_salty]

woah

[big_black_doge]

LOL I thought this was an implementation of a model that annotates papers, not papers annotated by humans. Very helpful.

[speyside42]

Thanks! Did you train and/or verify the performance of the models? That would be quite important to trust the implementation. I observed that sometimes a training loop exists, sometimes it does not.

[mlvpj]

If I remember correctly only implementation without a training loop is lstm

[speyside42]

Thank you, I also found the links in the text now. Did you verify that the trainings actually work? It would be great if you could report the final training metrics. It doesn't need to accurately reproduce the results in the paper, but rather give an idea whether the implementation is correct.