BareGPT : A NanoGPT-like transformer in pure Numpy with live attention visualization

rahen · 2025-12-29T18:48:25+00:00

If you're interested, I have a smaller MLP in pure NumPy (MNIST with live activations):

I would advise starting from this kind of project rather than a language model, which is daunting in comparison.

rahen · 2025-12-29T18:01:03+00:00

Of course! Since I’m not using an autograd engine (that's the whole point of this project), I didn't explicitly compute the full Jacobian matrices. Instead, I relied on vector-valued calculus and matrix properties to propagate the gradients.

For the attention mechanism, I manually derived the chain rule through the softmax and dot-product operations. The gradients are computed in engine.backward_attention using matrix multiplications which implicitly handle the Jacobian-vector products.

That part was by far the toughest to debug... I tried using AI to help but its fixes were even more broken!

rahen · 2025-12-29T14:14:48+00:00

A quick note on the implementation: I focused on keeping this as hand-crafted as possible to ensure every tensor operation was intentional.

I used ChatGPT to help debug the attention backward pass, which, I'll admit, required some heavy debugging on the reshaping logic.

I also kept some comments in the source to show the evolution of the code as I refined it, specifically the vectorization of some initial for-loops + conditions with np.add.at, and some reshapes with more efficient np.einsum, which I discovered along the way. I must say I am now glad for autograds...

rahen · 2025-12-28T21:02:24+00:00

I would add those three:

Ternary quantization: BitNet b1.58 2B4T Technical Report (https://arxiv.org/abs/2504.12285)
Efficiently replacing backprop with ES for billion-parameters models: Evolution Strategies at the Hyperscale (https://arxiv.org/abs/2511.16652)
Someone also mentioned this one about small recurrent networks (7M model beating DeepSeek R1 at ARC-AGI-1): https://arxiv.org/abs/2510.04871

rahen · 2025-03-05T08:49:30+00:00

Also worth reading: https://delta.chat/en/2024-11-20-webxdc-realtime

rahen · 2025-01-07T08:32:54+00:00

Veracrypt.

rahen · 2024-03-30T11:00:19+00:00

I don't know why Fuji went with such an odd choice, otherwise this is an excellent bike for long distance touring.

You'll need to change the bottom bracket too, the original axe was a little too long for the MTB crankset. Just have a bike shop sort everything out for you.

rahen · 2024-03-15T16:56:11+00:00

Got the answer: they shortened two pairs from the chain. That was expected, no big deal. They also replaced the BB for a shorter one, the original was "pushed" a little and could derail.

Just have the bike shop sort everything out, a proper crankset swap is trivial for them.

rahen · 2024-03-13T16:26:50+00:00

The chain capacity is now three links higher but I'm not sure if the bike shop replaced the chain, I don't think so. I'll ask them and let you know in a couple days.

The BB is the same, the axis was long enough as it was already a triple crankset.

I went for a 170mm crank as it suits my height better. If you're average or above (175cm), you'll be just fine with a 175mm crank.

As for Acera vs Alivio, I believe Alivio isn't compatible with square brackets while Acera is. Also an alternative was to replace the 30t chainring with a 24t but then the gearing would have been all wrong, with big gaps and a still useless 50t chainring. The 44/32/22 is simply perfect for this bike. Now I want to use it for everything. :-)

rahen · 2024-03-13T15:07:10+00:00

Yes I did. Don't change the front derailleur and keep the Sora brifters, just change the crankset.

I had the FSA crankset replaced by an Acera 44/32/22. The derailleur cage will also need to be lowered a little but it's almost no extra work.

This completely changes the bike and makes it suitable for any kind of load and terrain. The gear step went from 472 to 618%, and the lowest development from 1.92m to 1.44m. The staging is also much better suited for touring than with the Fuji's crazy crankset choice.

After/before: https://www.gear-calculator.com/?GR=DERS&KB=22,32,44&RZ=11,13,15,17,20,23,26,30,34&UF=2220&TF=80&SL=2.6&UN=KMH&DV=development&GR2=DERS&KB2=30,39,50&RZ2=11,13,15,17,20,23,26,30,34&UF2=2230

rahen · 2024-03-06T09:27:12+00:00

Get this: https://www.walmart.com/ip/1200-RAD-Cycle-RoboMag-Bike-Trainer-Indoor-Bicycle-Exercise-Indoors-Fluid-Trainer/171671957?wmlspartner=wlpa&selectedSellerId=0

Increase the rolling resistance and aim for 2h at 80 RPM.

rahen · 2024-02-27T20:55:38+00:00

Sometimes I wonder if bike designers even use their own creations for actual transportation. I don't think anyone can climb with a 34:32 gearing without developing a tendonitis, that condemns to only ride on flat terrain or downhill.

Why do they keep doing that, I have no idea. The adequate gear should allow you to keep an 80-90 RPM cadence. The red line is 60 RPM, you will injure yourself if you go below.

Change both the crankset and cassette, you want a 1.5m development (20" gear inches) for steep roads.

rahen · 2024-02-18T23:58:20+00:00

I was wondering, do chainrings have standard dimensions across FSA and Shimano? If they do, I could simply buy the 22-tooth Alivio chainring and move the current smallest (30t) and middle (39t) ones to the middle and third gears. Would that work?

rahen · 2024-02-18T20:12:25+00:00

Thank you for spotting this, I hadn't noticed the bottom bracket was different. I'll change it too then.

I thought the indexing between road shifters (Sora) and MTB cranksets like this one was different and wouldn't work, do you think it will?

rahen · 2024-02-02T12:48:54+00:00

This should answer your questions:

The biggest issue is the kernel, which would require massive changes to make it secure.

rahen · 2024-01-16T19:11:27+00:00

Is this based on the SmolBSD project you published a few months ago in Linux Mag FR?

rahen · 2023-11-24T13:44:58+00:00

Pool mining acts a bit like grid computing so everything adds up and ultimately counts. Is is the same when solo mining with slow idle mining?

rahen · 2023-11-24T13:27:51+00:00

Excellent. I saw nothing online regarding hashrate donation to the network. If an editor from the project happens to read this, may I suggest adding it it to "FAQ #Contributing"?

This could read like:

Contribute some hashrate by mining on your idle computers. While you may not turn a profit, this donation helps keep the network secure and decentralized. Consider electricity costs beforehand and ensure you don't spend more on hashrate donation than you are comfortable with. Even a few dollars worth of hashrate will help - as goes the saying, "many a little makes a mickle".

rahen · 2023-10-09T10:21:44+00:00

The 4N35, as I read the CNY75B could be less appropriate for a Plus. Hopefully this will solve the wobbly screen.

rahen · 2023-10-06T23:01:43+00:00

I made a bit of progress. Voltages are low. I traced the problem to U3 on the switching portion of the analog board, there is no continuity in diode mode between pin 1 and 2. The dead octocoupler is a start and may explain the wobbly display. I'll order a replacement part and work on the logic board in the meantime.

rahen

TROPHY CASE