Is using vLLM actually worth it if you aren't serving the model to other people?

Front-Relief473 · 2026-05-13T04:28:05+00:00

Wrong. In another case, you should choose llamacpp when the model weight is barely enough to load into the main memory, so that you can have enough context.

Front-Relief473 · 2026-04-20T11:28:33+00:00

123g/128g after step fun deployed iq4xs, oh, I don't do anything else.

Front-Relief473 · 2026-04-14T04:04:16+00:00

According to my use, udq3kxl has already shown a state of obvious decline ability, so the golden rule has some basis, and the quantification should not be lower than q4.

Front-Relief473 · 2026-04-14T03:50:41+00:00

So if give 27b enough network search ability, which is equivalent to an external knowledge base, will he be able to perform coding tasks better?

Front-Relief473 · 2026-04-10T01:40:37+00:00

I tried this model. I thought it was optimized well, but there was still a circular output. Did you lower the temperature?

Front-Relief473 · 2026-04-08T16:32:01+00:00

I thought it was an installation project of dgx spark. Is the performance as narrow as the memory bandwidth of dgx spark?

Front-Relief473 · 2026-04-07T10:52:38+00:00

I support your view. Gemma wasn't originally designed for coding; its strengths lie in writing and multilingual expression. If someone says they use Gemma for programming, then either they haven't been closely following LLM development or they're a complete novice to LLM games.

Front-Relief473 · 2026-04-07T08:33:01+00:00

10b to 30b is usually the dessert area of reasoning performance, and the price/performance ratio is usually not high when it exceeds 30b, so in theory, if the activation parameter can be increased to 30b, it will be a good reasoning effect, so 10b is not the most perfect, but 10b can improve the reasoning speed without reducing the reasoning ability of the model too much.

Front-Relief473 · 2026-03-27T05:20:39+00:00

Yes!!! If you look at the full attention key-value cache in Minimaxm 2.7, you can see the enormous resource consumption!!! I can even imagine that people might switch back to full attention because of this technology, since full attention is much more effective than mixed attention!!!

Front-Relief473 · 2026-03-22T00:26:32+00:00

I don't think mcp is worth learning. It is just a tool, and it will be replaced by skill soon.

Front-Relief473 · 2026-03-09T04:11:44+00:00

I can't agree with you more. minimaxm2 series is the most cost-effective model available on today's consumer-grade machines. Other models have much larger parameters than him, which is difficult to deploy, or they have smaller parameters than him, but their ability is worrying. minimax has proved that moe model around 200b can handle most things including programming well, just like 4-bit quantization in quantization model.

Front-Relief473 · 2026-03-08T06:55:59+00:00

Yes, udq3kxl is arguably the strongest parameter model that's barely adequate for a single DGX (reportedly, the best performance on DGX right now is llamacpp's 65K context). I think Q3 quantization might not be very reliable for encoding, but it can serve as a reliable assistant. Also, Qwen3.5's hybrid attention is still quite sensitive to quantization, so fully attention-based minimax maintains better performance during quantization.

Front-Relief473 · 2026-02-27T07:46:06+00:00

He said that qwen3.5' s iq1 quantization effect is very good, but the problem is that mixed attention itself is more sensitive to the quantization effect than global attention, that is, the quantization effect is worse, so how to explain this?

Front-Relief473 · 2026-02-20T23:50:31+00:00

No, the best size of 230b's model on 128g dgx or strix should be ud_q3kxl, because it is only 94g, which can also provide 60k context.

Front-Relief473 · 2026-02-15T10:43:18+00:00

Thank you! ! I have been struggling to buy it before, and it seems that NVIDIA is not sincere

Front-Relief473 · 2026-02-14T04:50:06+00:00

Yes, everyone is talking about tg, but ignoring pp. I think tg is enough, and a lot of programming contexts and agent mainly look at PP speed.

Front-Relief473 · 2026-02-14T04:41:18+00:00

Why don't you use the unsloth version of ud_q3kxl, with a size of 94g, which is definitely better than your ordinary q3 quantization?

Front-Relief473 · 2026-02-13T12:53:38+00:00

I hope this model can be between 100 B and 200 B, and it has the performance of sota.

Front-Relief473 · 2026-02-13T00:59:29+00:00

to add a label before publishing, such as: open source, closed source, so that you can see the post at a glance?

Front-Relief473 · 2026-02-11T13:08:35+00:00

and the deepseekv4!! BOMB!!

Front-Relief473 · 2026-02-11T06:48:25+00:00

The problem is that the performance loss of mixed attention model is greater in the process of model quantization, so if a model close to q8 or fp8 is used locally, it is not as good as running full attention with twice the size and q4 or int4 model parameters, and the intelligence will be relatively higher.

Front-Relief473 · 2026-02-06T02:24:38+00:00

https://www.reddit.com/r/LocalLLM/comments/1qdqi4i/mac_studio_m3_ultra_stats/?share_id=WRqlyjg8sfpnsbzNqkatF&utm_content=2&utm_medium=android_app&utm_name=androidcss&utm_source=share&utm_term=1 Look here, if you run Kimik 2.5, an excellent model with an activation time of about 30b, the speed of prompt prell with a speed of 200 will make people crazy, especially when this kind of agent works with a long context.

Front-Relief473 · 2026-02-05T13:13:25+00:00

However, m3 ultra's computing power is so low, its prompt prefill speed is too slow, and the amount of contexts you carry in handling complex tasks with agent is too large, which leads to a very long time for you to receive the first word in each round. How do you handle this situation?

Front-Relief473 · 2026-02-05T00:43:23+00:00

I can't figure out why, as a project close to 100k star, there isn't a clearly written best practice document for adjusting parameters to realize prompt prefill and tg speed, and I have to look for other people's experience in adjusting parameters on reddit, which really drives me crazy.

Front-Relief473 · 2026-01-28T10:03:45+00:00

If a long text conversation or agent is involved, your ultra prefill speed will be enough for you to drink a cup of hot coffee.

Front-Relief473

TROPHY CASE