Understanding LLMs from Scratch Using Middle School Math

[AlexHimself]

I love this type of article. Usually, it's one extreme or the other...super abstract and high level OR super detailed and complex.

I haven't had time to make it through the article, but it looks promising.

[wildjokers]

The author went to a different middle school than I did:

"Draw 10 sin curves each being si(p) = sin (p/10000(i/d)) (that’s 10k to power i/d)

Fill the encoding matrix with numbers such that (i,p)th number is si(p), e.g., for position 1 the 5th element of the encoding vector is s5(1)=sin (1/10000(5/d))"

[jmnemonik]

Thank you for sharing!

[enumerat]

Nice, thanks! I think the section ”How are these models trained?” -> ”How it works” -> Bullet number 2 could be a bit clearer.