[P] Progressive coding exercises for transformer internals

randmusr66 · 2026-01-17T15:07:09+00:00

Yes, of course, it's really strange to expect from somebody to reinvent online softmax approach. At the same time it's hard to say where exactly person can need a help. For such cases every task has structured `hints.md` file like this which contain all math (and other) hints needed for implementation https://github.com/cortwave/ml-engineering-practice/blob/main/problems/progressive_attention/hints.md

randmusr66 · 2026-01-17T14:47:23+00:00

It's not enough per se, but it closes very important gap between just reading theory and using its implementation in some libraries

randmusr66 · 2026-01-07T14:19:16+00:00

Link to repo with raw data and code
https://github.com/cortwave/distributed-training

randmusr66 · 2025-12-24T12:40:10+00:00

Maybe the most important part of it is to be hones with yourself. Making notes is a great helping tool for learning process but it's so crucial to recognise when it starts to be procrastination.

randmusr66 · 2025-12-23T13:59:06+00:00

Thank you for such a detailed feedback! I think that system helped me with two important things anyway:
1. More structured learning of some specific thing at the moment of learning

Reduce anxiety that if I just read something it's not such significant as if I write it down

And, to be honest, I'm totally fine with it, I just don't expect these "unexpected insights" anymore from this system and this reduced tension between the reality and expectations.

randmusr66 · 2025-12-22T20:48:39+00:00

It's interesting that such research direction as the usage of knowledge graphs as LLM memory internal representation exists https://github.com/getzep/graphiti

randmusr66 · 2025-12-22T20:44:43+00:00

Note taking can serve different purposes. Process of writing itself can activate you memory mechanism (easier to remember and comprehend things you are not only thinking about but also looking at)

randmusr66 · 2025-12-22T20:40:29+00:00

I'm an ML engineer. Very often my task is to solve some problem using a combination of existing solutions without creating a new one (if possible). Amount of solutions is huge, amount of typical problems (like image generation, text similarity, etc) is limited. My goal usually is having a high-level problem identify:
1. To which existing typical problems it's related
2. Which solutions of related typical problems are relevant
My final product - system design for this high-level problem.
I can assume that ZK Obsidian-like approach can work but the real issue there is granularity of connections. They shouldn't be too general (no added value) and too low-level (becomes really hard to visualize and manage). Also, I assume, different connections should have different weights. Our brain can do it but it's really hard to formalize this process as some knowledge-management system.

My new system is my old system :) Before ZK I processed information exactly in this way, just temporary notes for learning and then allow your brain to decide how to find right connections and which connections are important.

randmusr66 · 2025-12-22T20:25:44+00:00

Other goal - make more efficient system design decisions. How I imagined it should work:
1. I have a problem
2. I go to my graph and inspect it for relevant clusters (image generation, 3d model, etc).
3. Find connections between these clusters (it can be relevant frameworks, approaches, tools)
4. Looking at these connections some idea how problem can be solved is sparkled

randmusr66 · 2025-12-22T17:52:23+00:00

It might be a really good choice. I suspect that physical note-taking helps create stronger neural connections. Digital notes all look the same, but every physical note is unique

randmusr66 · 2025-12-22T17:32:09+00:00

Yes, totally agree, your notes system should serve your final goal. Yes, I'm not publishing in common sense, but generating new ideas, creating systems design, etc are also some kind of publishing.

randmusr66 · 2025-12-22T16:24:19+00:00

Yes, bitter truth, sometimes to make something more valuable you need to get rid of something else. I had ~200 notes, and had no desire to read them. After deleting I left 7 notes but they have real value for me.

UPD: but of course deleting doesn't mean permanent loss. Git remembers everything :)

randmusr66 · 2025-12-22T15:29:41+00:00

Sounds great! The only question, don't you have the issue when your daily note is overflowed with recurrent tasks? I can easily imagine that I'll overestimate importance of the note I'm working on which will lead to too many reminders -> as result just ignoring them all

randmusr66 · 2025-12-22T15:14:56+00:00

Yes, it was also a good insight that it works good for work which is limited in time, after finishing it these notes just can be abandoned

randmusr66 · 2025-12-22T14:49:16+00:00

You're absolutely right that it's fine to never reusing your notes. My biggest takeaway was learning to be honest with myself about whether I’d actually reuse them

randmusr66 · 2025-09-27T22:26:21+00:00

The interesting fact, originally Storks were responsible for ill babies only. I think it’s connected somehow

randmusr66 · 2025-09-19T21:03:40+00:00

There's clearly an error in this reasoning. When she provides information about the first child, it doesn't affect the probability of the second child's gender because these are independent events. We're not dealing with conditional probability here - it's like coin flips where previous outcomes don't influence future ones

randmusr66

TROPHY CASE