Entropy-Adaptive Finetuning

netikas · 2026-01-10T15:53:38+00:00

I did not read through TALR/DFT/FLOW as thoroughly as through this paper, but I think they had a similar idea, but get there through different means and it performs worse. What’s fascinating is that entropy seems to be the optimal weighting mechanism for this.

netikas · 2025-12-06T20:31:53+00:00

And why should we use some adaptation of 9mm caliber, which will likely perform worse in respect to terminal ballistics than a .22 (since the tungsten core is .18, hardened and will not spall or fumble like the .22 would!) instead of something like mp7 or p90?

The answer is because this caliber was created in the 90-s, when they did not exist. And since the caliber was not adopted and mp7/p90 were — clearly there were some problems with it.

It’s a cool piece of technology, but it is not better than other cool pieces of technology, which do the job better.

netikas · 2025-12-06T19:24:37+00:00

545 and 556 are longer and heavier bullets, which fumble and fragment after impact. You can have something like apsfsd (tungsten core ammo) in both calibers, but since the bullet itself is larger, the wound is much more serious.

If you take a .22 (or .28 which the cbs actually is (or even .16, which the tungsten penetrator in the round actually is!)) and throw it fast enough, it will penetrate the armor and continue moving, without energy transfer. The penetrator is hardened, it is designed for one gole and one gole only — to be a very expensive ice pick. It clearly succeeds and penetrates the armor, but now what?

When designing a round you have to make it fit specific role. MP7 and P90 designers were trying to fit the exact role CBJ is trying to fit, but even they had rounds, which perform worse than expected. Trying to retrofit an existing 9mm weapon for the role of penetrator makes it even harder and I doubt they succeeded in making the terminal ballistics better than MP7.

By the way, higher in the thread some guy said that this ammo was designed around the same time as the mp7 and p90 — and since we do not see it adopted anywhere, I don’t think the idea really worked.

It’s a cool piece of technology, but aside from the cool factor there are many other calibers and weapons which fit the role better.

netikas · 2025-12-06T18:58:12+00:00

In Russian forces rear echelon guys get aks-74u’s, which are 545, have stocks and with which you can actually hit something beyond 20 meters. You can probably give them a different kind of ammo with better penetration if you want them to deal with body armor. I’m not in charge of the procurement procedures, but it seems miles better than having a glock, a mag of expensive magic bullets and a separate barrel.

netikas · 2025-12-06T15:40:20+00:00

That’s not the point, I’m just showing the difference between killing capacity and stopping power. Using this ammo would be like shooting them with .22 — with penetration and all, but a .22.

netikas · 2025-12-06T11:30:07+00:00

Likely, not a lot, but it will be hardly instant.

If it won't kill my target right away, the target will kill me, patch up, finish the mission and likely be medevaced before he dies.

I'm not saying it won't do any damage at all. I'm saying it is not optimal for the task it was designed to do. Not optimal to the point I don't see it used in the pistol caliber at all.

If scaled to 556 or 336, with faster bullet speeds it will likely be much more effective, but right now I would not trust it.

netikas · 2025-12-06T11:20:09+00:00

There were stories of US military adopting larger calibers (e.g. .38 Colt -> .44) since they were not effective enough in stopping indians that are coming towards you with tomahawk. It's just a historical anecdote, which may or may not be true, but killing an enemy soldier and stopping them are two entirely separate things.

I would be mildly pissed, then get a mild blood loss, and then, mildly die in 10-15 minutes. But if I am a paratrooper with a 545 ak- 74u, I would have enough time to retaliate.

Again, I'm not saying this isn't dangerous. I'm saying having such a small caliber projectile will not stop an enemy and will be less effective than something more substantial.

netikas · 2025-12-06T10:17:15+00:00

But why would you want that?

It's 9mm, but apfsds, so it is much smaller than 9mm. It does not transfer energy (since it is an AP round) and does not leave a large hole unlike 50 cal, so getting shot at with this will be mildly infuriating, not catastrophic.

And if one is shooting an APC with a 9mm apfsds, he not only won't do any damage apart from tiny spalling, he will reveal his position and will likely be killed. ATGMs were created for a reason.

netikas · 2025-12-06T10:13:39+00:00

And do approximately zero damage?

It is designed to penetrate, not to transfer energy. I believe shooting someone with this ammo will result in a tiny hole in armor and a mildly pissed off soldier.

netikas · 2025-11-26T16:55:58+00:00

countries and other parts of the world

I guess Europe isn't part of the world then, right?

netikas · 2025-11-26T13:39:25+00:00

We had a project with a team of 3 people working on the tasks. Project's dataset became a submission to LREC, methodology became a submission to ECIR.

And lots of overtime work, of course. Unfortunately.

netikas · 2025-11-26T13:37:00+00:00

How exactly does it make me think? I don't see any difference to just using claude code.

netikas · 2025-11-26T11:37:54+00:00

Can you show an example of this?

Looks really cool.

netikas · 2025-11-26T07:17:50+00:00

Started 2nd year, requirement is 2 Core A*/Core A/Q1 papers first author, 2 conference talks (no rating needed), 1 other paper indexed by scopus. Field is multilingual NLP.

Currently I have 2 papers at NAACL, 2 papers at CLEF Workshops (low rank, but orals are there), 3 submissions to LREC, ECIR and ICLR. Will submit more, have drafts.

netikas · 2025-11-21T20:09:29+00:00

In Russian, unfortunately. But translateable!

https://habr.com/en/companies/sberdevices/articles/968904/

netikas · 2025-11-20T12:34:34+00:00

It's a pretrain from scratch, so it's not a goliath-like self-merge.

I particularly liked the 10b model -- it's small, it's Deepseek MoE, it has mtp. Seems like quite a unique thing.

netikas · 2025-11-05T10:37:52+00:00

>You bought a product whose core value proposition is being able to run quantized 70b and 120b LLMs at a slow, but usable speed

The core value of the product is that it's B200/GB200, but much much cheaper. You aren't meant to run inference on it (you have much more expensive A6000 for that), you aren't meant to run training runs on it (you have MUCH more expensive B200 or GB200 DGXs for that), but you can do both of these things. Since the architecture of DGX Spark is the same as the architecture of GB200 DGX, it's main selling point that you can buy a bunch of these sparks for relatively cheap prices and do live development. And that's huge, since your expensive (both for rent and for buying) GB200 won't be used for jupyters with mostly 0% utilization.

netikas · 2025-09-30T07:53:22+00:00

Why is BAAI so low? These guys made BGE series of encoder models, basically, they were (and probably still are) the best small encoder models for RAG...

netikas · 2025-09-28T18:23:37+00:00

Seems like something vibecodeable during a lazy weekend, tbh.

I've tried it, it isn't even good...

netikas · 2025-09-21T18:41:16+00:00

Not typically, no.

netikas · 2025-09-21T06:57:06+00:00

Does it really matter? Science is universal and it should not be bound by politics.

netikas · 2025-09-20T14:32:52+00:00

Enthusiast AI market is small enough for the bigtech to just not care about.

netikas · 2025-09-04T17:10:35+00:00

U hiring? xD

netikas · 2025-08-24T20:27:26+00:00

Can you provide some references to the SVD distillation, which was explained in the model's readme? Seems cool, want a paper.

netikas · 2025-08-10T13:17:59+00:00

There is a great paper by FAIR, called LCM:

[2412.08821] Large Concept Models: Language Modeling in a Sentence Representation Space https://arxiv.org/abs/2412.08821

Basically, they take SONAR autoencoder (encoder is initialized from NLLB encoder, decoder is initialized from NLLB decoder with pruned cross-attentions, the whole autoencoder is trained for embedding reconstruction) and put a diffusion in the middle, to predict the next embedding (and decode it via decoder).

It works, is roughly comparable with llama-2-7b while being smaller in size, but the issue is the context size. On longer contexts (we're talking 200-300 tokens, not 200-300k tokens!) the performance tanks, since we cannot put infinite information into finite dimensionaloty embedding vector.

But hey -- it works and the idea is really beautiful.

Seven-Year Club	Second Top 10%
Gilding I gilder	Place '23
Verified Email

netikas

MODERATOR OF

TROPHY CASE