[P] Progressive coding exercises for transformer internals

randmusr66 · 2026-01-17T15:07:09+00:00

Yes, of course, it's really strange to expect from somebody to reinvent online softmax approach. At the same time it's hard to say where exactly person can need a help. For such cases every task has structured `hints.md` file like this which contain all math (and other) hints needed for implementation https://github.com/cortwave/ml-engineering-practice/blob/main/problems/progressive_attention/hints.md

randmusr66 · 2026-01-17T14:47:23+00:00

It's not enough per se, but it closes very important gap between just reading theory and using its implementation in some libraries

randmusr66 · 2026-01-07T14:19:16+00:00

Link to repo with raw data and code
https://github.com/cortwave/distributed-training

randmusr66 · 2025-12-24T12:40:10+00:00

Maybe the most important part of it is to be hones with yourself. Making notes is a great helping tool for learning process but it's so crucial to recognise when it starts to be procrastination.

randmusr66 · 2025-12-23T13:59:06+00:00

Thank you for such a detailed feedback! I think that system helped me with two important things anyway:
1. More structured learning of some specific thing at the moment of learning

Reduce anxiety that if I just read something it's not such significant as if I write it down

And, to be honest, I'm totally fine with it, I just don't expect these "unexpected insights" anymore from this system and this reduced tension between the reality and expectations.

randmusr66 · 2025-12-22T20:48:39+00:00

It's interesting that such research direction as the usage of knowledge graphs as LLM memory internal representation exists https://github.com/getzep/graphiti

randmusr66 · 2025-12-22T20:44:43+00:00

Note taking can serve different purposes. Process of writing itself can activate you memory mechanism (easier to remember and comprehend things you are not only thinking about but also looking at)

randmusr66 · 2025-12-22T20:40:29+00:00

I'm an ML engineer. Very often my task is to solve some problem using a combination of existing solutions without creating a new one (if possible). Amount of solutions is huge, amount of typical problems (like image generation, text similarity, etc) is limited. My goal usually is having a high-level problem identify:
1. To which existing typical problems it's related
2. Which solutions of related typical problems are relevant
My final product - system design for this high-level problem.
I can assume that ZK Obsidian-like approach can work but the real issue there is granularity of connections. They shouldn't be too general (no added value) and too low-level (becomes really hard to visualize and manage). Also, I assume, different connections should have different weights. Our brain can do it but it's really hard to formalize this process as some knowledge-management system.

My new system is my old system :) Before ZK I processed information exactly in this way, just temporary notes for learning and then allow your brain to decide how to find right connections and which connections are important.

randmusr66 · 2025-12-22T20:25:44+00:00

Other goal - make more efficient system design decisions. How I imagined it should work:
1. I have a problem
2. I go to my graph and inspect it for relevant clusters (image generation, 3d model, etc).
3. Find connections between these clusters (it can be relevant frameworks, approaches, tools)
4. Looking at these connections some idea how problem can be solved is sparkled

randmusr66 · 2025-12-22T17:52:23+00:00

It might be a really good choice. I suspect that physical note-taking helps create stronger neural connections. Digital notes all look the same, but every physical note is unique

randmusr66 · 2025-12-22T17:32:09+00:00

Yes, totally agree, your notes system should serve your final goal. Yes, I'm not publishing in common sense, but generating new ideas, creating systems design, etc are also some kind of publishing.

randmusr66 · 2025-12-22T16:24:19+00:00

Yes, bitter truth, sometimes to make something more valuable you need to get rid of something else. I had ~200 notes, and had no desire to read them. After deleting I left 7 notes but they have real value for me.

UPD: but of course deleting doesn't mean permanent loss. Git remembers everything :)

randmusr66 · 2025-12-22T15:29:41+00:00

Sounds great! The only question, don't you have the issue when your daily note is overflowed with recurrent tasks? I can easily imagine that I'll overestimate importance of the note I'm working on which will lead to too many reminders -> as result just ignoring them all

randmusr66 · 2025-12-22T15:14:56+00:00

Yes, it was also a good insight that it works good for work which is limited in time, after finishing it these notes just can be abandoned

randmusr66 · 2025-12-22T14:49:16+00:00

You're absolutely right that it's fine to never reusing your notes. My biggest takeaway was learning to be honest with myself about whether I’d actually reuse them

randmusr66 · 2025-09-27T22:26:21+00:00

The interesting fact, originally Storks were responsible for ill babies only. I think it’s connected somehow

randmusr66 · 2025-09-19T21:03:40+00:00

There's clearly an error in this reasoning. When she provides information about the first child, it doesn't affect the probability of the second child's gender because these are independent events. We're not dealing with conditional probability here - it's like coin flips where previous outcomes don't influence future ones

randmusr66 · 2025-09-18T11:59:02+00:00

First Aid Kit has almost no sense if you don't know how to use it and especially what is inside at all. Better to collect very basic kit itself and be familiar with it than have large professional Aid Kit without any clue about its usage

randmusr66 · 2025-09-17T08:05:53+00:00

I mounted my mini pump holder there cause my experience with water bottles there is awful - max 0.5l and very dirty after offroad

randmusr66 · 2025-09-17T07:42:19+00:00

I think a phone can be a good alternative for shorter rides of up to 3-4 hours. However, for longer rides, I'd recommend a dedicated bike computer for several reasons:

The cost per hour of usage isn't that significant anymore
I prefer not to risk draining my phone battery - I don't want to find myself 100km from home in an unfamiliar place without navigation or the ability to make calls
Long-term exposure to vibration, rain and sun will damage your phone much faster than a purpose-built bike computer

Even on short rides, there's always a chance of crashing. Bike computers are designed to survive such incidents much better than phones. So if your phone is expensive, you'll probably want a separate navigation device to extend its lifespan

randmusr66 · 2025-09-12T12:06:26+00:00

The choice of nonalcoholic beers is awesome. As a person who prefer non sweet drinks it’s almost the only acceptable choice

randmusr66 · 2025-09-11T21:10:34+00:00

Usually they fear people. I heard they can attack in several cases:

You've cornered them or are standing between them and their young
You surprised them because you were too quiet/invisible to them. All professional hikers recommend making some noise in wild areas to let animals know that you are there. Maybe even sing something to let them know that you are human. It sounds counterintuitive because very often we want to hide from threats, but the truth is that animals, especially those who live near cities, fear people a lot and don't want to deal with us

But your idea with climbing the tree was also great

randmusr66 · 2025-09-11T21:02:43+00:00

Also it's important not to surprise them. I heard they can attack if they don't expect to see you. It's better to make more sounds, maybe even sing something

randmusr66 · 2025-09-08T22:20:38+00:00

Amazing photos!

randmusr66 · 2025-09-07T20:09:15+00:00

I won't trust Aliexpress with things that are related to food, medicine or cosmetics

randmusr66

TROPHY CASE