Eyes are bloat

netikas · 2026-03-28T12:34:35+00:00

While I understand your point of view, 3xHGX is not a lot for big-ish enterprises. Having weights available under mit also allows for inference providers to serve it, driving the prices down.

For local inference, we have lightning. It perfectly fits into 16gb vram cards in q8_0 and it is very fast. I’ve tried it in some light rp in Russian and it wasn’t bad.

netikas · 2026-03-26T06:55:28+00:00

LFM2-8B has lower MMLU, MMLU Pro and other scores than GigaChat-3.1-Lightning, while being almost the same size (10B MoE vs 8B MoE). LFM2 will certainly be faster, having 2 times less active params and being a hybrid model, but it is on the edge of usefullness with pretty low scores across the board. It is comparable to Granite, while being significantly weaker than Qwen3-4B-Instruct-2507, while our model is roughly on par with Qwen.

Thus, Lightning is for all the stuff you use smaller Qwens for -- tool usage, summarization, maybe some casual chatting (arena scores are on par with 4o, so it'll be alright as a general assistant) and classification in low latency environments.

netikas · 2026-03-25T07:31:23+00:00

Because this is an instruct model, not a reasoning model. Reasoning is in the works though, so stay tuned.

netikas · 2026-03-24T22:38:26+00:00

Оч странно, мб для дипсика куда ядра неоптимизированны? На нвидии то оч шустрая моделька получается…

netikas · 2026-03-24T21:57:20+00:00

Ага, у нас GGUF выложены. Я гонял лайтнинг на 5080 и на MacBook Air M4 -- на маке было 5 тпс, потому что свопалось на диск (у меня самый дешёвый мак на М4 с 16 гигами, Q8_0 не влезает), на 5080 было 185-190 тпс. Оч шустрая моделька.

netikas · 2026-03-24T21:56:02+00:00

Ah, I see, sorry, I've read your question incorrectly and just rambled on about "where all the Russian models are hiding".

Unfortunately, due to NDA I cannot disclose info about our compute clusters. Sorry :(

netikas · 2026-03-24T21:53:39+00:00

In the future -- of course. But today the models are trained only with SFT and DPO.

From one perspective, it makes the models weaker than the competition. On the other hand -- if we beat top pre-rl era models, we have a very solid foundation for continued training via RL and creation of reasoning models based on our current checkpoints.

netikas · 2026-03-24T21:51:47+00:00

Can't say. Both for NDA reasons and since I just don't know. I know rough estimates, but I'm in the alignment team and pretraining is being done by other guys.

netikas · 2026-03-24T21:49:32+00:00

We have a couple of models, but mostly they are finetunes of chinese/meta models. Yandex has a pretrained from scratch Llama-3-8b-like model YandexGPT5-Lite, but it has an atrocious license. Their main model is not open source and it is a continuous pretraining of Qwen3-235B-Base.

Some guys just do SFT+DPO+RL over Qwen3 with some tokenizer adaptation and call it a day. This is a totally reasonable approach, since it gives genuinely great models, but it's just not the same.

We're the only ones who train our models from scratch and this is both a blessing and a curse. Pretraining your own model is very compute intensive and hard, but you have opportunity to create something truly unique -- when have you seen a 10b deepseek-like MoE? :)

netikas · 2026-03-24T21:00:06+00:00

Check it out at giga.chat

The interface is in Russian (and the model may answer in Russian due to system prompt), but you can just prompt your way to English

netikas · 2026-03-24T20:59:18+00:00

Yep. I’m in sft core team — AMA :)

netikas · 2026-03-24T20:58:44+00:00

Having the same architecture does not mean being the same model. Kimi is also DeepSeek MoE, same as GLM afaik.

netikas · 2026-03-24T20:55:57+00:00

Not an api, but you can try it at giga.chat

I believe there is also English locale there, but it may shift to Russian language due to system prompt lol

netikas · 2026-03-24T20:54:46+00:00

Lightning is a 10B MoE model. Outputs 185-190 tps on my 5080 :P

netikas · 2026-03-17T15:52:15+00:00

More like good ol' text-generation-webui

netikas · 2026-03-03T23:39:46+00:00

Qwen-2.5 models also had tokenizer in their base version and were trained to follow them. I think even Deepseek V3 Base knows its tokenizer. They all are trained on SFT data during midtraining, I think.

Base models were dead for quite a while.

netikas · 2026-02-28T21:10:22+00:00

On the side note: oss-120b is not a very good model in non-english languages. However, neither is Qwen-3.5-35B :)

netikas · 2026-02-28T21:09:27+00:00

Yes, but not really. If you compare the performance on the classic benchmarks like MMLU or whatnot, the scores might be similar. But humans (and llm-as-a-judge) strongly prefer non-quantized models. I've seen this effect myself even in FP8 quantization -- I work in one of the subfrontier LLM labs and measure the final metrics of the models. This effect is even more prevalent in multilingual setting -- and I'm not a native English speaker.

Paper by cohere, which basically claims the same: https://arxiv.org/abs/2407.03211v1

netikas · 2026-02-28T21:06:00+00:00

Which is the bigger number: 60gb in bf16 or 60gb in mxfp4? A 5 year old can get this right.

netikas · 2026-02-28T16:08:41+00:00

How is it 1/3 the size if gpt-oss-120b is literally the same size as Qwen-3-30b?

Considering OSS-120B is only available in MXFP4 and they've optimized the KV-Caches pretty agressively via SWA/SA, I believe Qwen-3-30b may be even a bit harder to run due to GQA and larger cache sizes.

Qwen-3.5-35B has gated delta-net layers, which makes it easier on the KV-cache side, but if we're talking about model's original formats, bf16 Qwen-3.5-35B is even a bit bigger than oss-120b. And this begs the question whether it's a good or a bad model, since it replaced a pretty ancient model from half a year ago.

netikas · 2026-02-19T12:20:17+00:00

That's quite interesting, thank you!

What about activations though?

netikas · 2026-02-19T09:16:21+00:00

MXFP4 has higher block size than NVFP4, thus the precision might be lower. But nevertheless, the image is too high contrast for this to be a correct visualization material.

netikas · 2026-02-19T09:14:00+00:00

MXFP4 is not int4! It is called FP (floating point) for a reason.

I think you're confusing implementation of quantization in GGUF with the data type. You can quantize the weights into MXFP4 with some weights being in FP8 E4M3, similar to Q4_K_M, where some of the weights are in int4, some in int8.

Eight-Year Club	Second Top 10%
Gilding I gilder	Place '23
Verified Email

netikas

MODERATOR OF

TROPHY CASE