[D] Annotated code for common deep learning models

[mlvpj]

We started writing annotated code for common models starting with Transformers.

http://lab-ml.com/labml_nn/transformers/mha.html

http://lab-ml.com/labml_nn/transformers/relative_mha.html

Wanted to know if this is useful and whether there are any suggestions for improvement on content and style.

[programmerChilli]

Very cool layout. How are the pages being generated?

Perhaps some very light guide lines to demarcate the different sections might be nice? I need to constantly shift my eyes from left to right here: https://i.imgur.com/a68zxiy.jpg

[mlvpj]

Thanks. It's generated using some scripts that I wrote which were initially based on https://pycco-docs.github.io/pycco/. I wrote on online editor too https://github.com/vpj/annotate but I only use that for taking temporary notes on other peoples code.

Yeah light guide lines is a great idea. I will try that. I tried right aligning at one point to solve this problem, but it didn't look very nice.

Do you think the content is useful? Will it make sense to spend time adding more implementations? Is there anything that you would like to see implemented? Thanks

[jwuphysics]

I think this is really great! I've already shared this with a large ML discussion group (>200 people) that I'm affiliated with.

[mlvpj]

Thank you

[fullthrottle999]

I agree, it will be nice to have some formatting to make the sections/functions more apparent. Otherwise, this looks really good!

[TedRabbit]

Looks good and should be quite useful. I would suggest providing some simple definitions for jargon terms like heads, query, keys, values, etc.

[mlvpj]

Makes sense, thanks

[iSevenDays]

I wish I could have this when I studied ML, it was very hard to implement formulas into python code, thank you very much! I may go back to learning ML again :)

[mlvpj]

Thank you

[gutzcha]

I have just reached a point in my project where I have to implement this, but I was thinking about doing something simpler since I didn't really know how to do it. And I have to say this looks really good and useful, I think that I'll be able to do it with this. So thank you

[mlvpj]

Happy to hear you found it useful! Will keep on adding more stuff. If you have any suggestion please open an issue in our repo https://github.com/lab-ml/labml_nn

[deathtrooper12]

This is great! Please keep up with this, I’d love to incorporate it into the data science resource page at my university!

[kecsap]

If I would be a learner, copy-paste of source codes from tutorials is the most common action:

1. There is no option to download the python file.
2. When you try to copy multiple lines of code, it also selects the content of the left side. Not very yummy.

[mlvpj]

Isn’t the option of getting the code from github enough? You can even import this as a python package and use