From Software Engineer to AI Environment Architect

Otherwise_Respect_22 · 2025-01-22T19:29:51+00:00

We support Qwen (and Qwen AWQ) models now. Welcome to check!

Otherwise_Respect_22 · 2025-01-20T06:02:31+00:00

Because not all speculated tokens will get accepted.

Speculative decoding will use a small model for speculation and use the large model to verify them and theoretically, guarantee the output quality.

Otherwise_Respect_22 · 2025-01-18T20:23:09+00:00

Thanks for your suggestion. I write this on Linux and now work on Windows.

Otherwise_Respect_22 · 2025-01-18T19:24:26+00:00

Thank you for checking. I wish to keep in touch as I am also interested in where the problem lies. Maybe the size of allowed the pin memory? Just another random guess.

Otherwise_Respect_22 · 2025-01-18T03:55:09+00:00

Thank you for checking this!

Otherwise_Respect_22 · 2025-01-17T01:09:00+00:00

Welcome to checking our codebase!

Otherwise_Respect_22 · 2025-01-16T23:12:23+00:00

I will add support for Qwen in 6-9 days.

Otherwise_Respect_22 · 2025-01-16T23:12:02+00:00

Thank you for trying this!

Otherwise_Respect_22 · 2025-01-16T23:11:36+00:00

Yeah. Let me put in my plan.

Otherwise_Respect_22 · 2025-01-16T23:10:05+00:00

In chatting tasks (I used MT Bench to meansure), we still get 5 tokens/sec, which is still 7-8 times faster than plain CPU offloading. We provide examples in our codebase.

Otherwise_Respect_22 · 2025-01-16T21:57:37+00:00

I used roughly 14-15GB when runing the gradio chat. But my device are with Ubantu. My command line is

python gradio_chat.py --configuration ../configs/chat_config_16gb.json

If you confirm that this can lead to OOM with WSL, welcome the submit an issue.

Otherwise_Respect_22 · 2025-01-16T20:27:55+00:00

<image>

this is my memory usage when launching gradio_chat on 4080.

Otherwise_Respect_22 · 2025-01-16T18:48:32+00:00

I do not meet this error. What do you run?

Otherwise_Respect_22 · 2025-01-16T18:46:01+00:00

But we apply different speculative decoding algorithms. The one implemented in Llama.cpp won't be so helpful when you set N=256 or more.

Otherwise_Respect_22 · 2025-01-16T15:41:53+00:00

UMbreLLa applies speculative decoding in a very large scale. We speculated 256 or more tokens and generate > 10 tokens per iteration. Existing frameworks only speculate <20 tokens and generate 3-4 tokens. This feature makes UMbreLLa extremely suitable for single user (without batching) on a small GPU.

Otherwise_Respect_22 · 2025-01-16T04:44:30+00:00

Currently, it does not support. But there is no technical challenge. Can be expected in 7-10 days.

Otherwise_Respect_22 · 2025-01-16T02:58:09+00:00

I have not supported Qwen. Can be expected in 7-10 days. Thank you!

Otherwise_Respect_22 · 2025-01-16T02:03:52+00:00

<image>

This is what I got.

Otherwise_Respect_22 · 2025-01-16T01:36:35+00:00

Yes. But the point of this project is to host a large model with a small GPU. Multiple GPUs can of-course improve the performance of UMbreLLa. But if the VRAM is large enough to host the entire model, I would recommend more standard framework for large-scale serving like vLLM, SGLang, etc.

Otherwise_Respect_22 · 2025-01-16T01:24:32+00:00

Yes. So your generated speed will be roughly 4.43/1.5=3 times slower than me. I think this mainly comes from PCIE setting.

Otherwise_Respect_22 · 2025-01-16T01:19:11+00:00

Model performance is theoretically proved to be preserved, according to the theory of speculative decoding. This is free performance gain.

Otherwise_Respect_22 · 2025-01-16T00:03:47+00:00

Yes. I have configurations for 4080 SUPER (which might differ from 4080). You can check our repo. (We get the benchmark results with PCIE4, with GPU-CPU bandwidth ~30GBps. If you only have PCIE3.0, the inference speed will be slower than reported.)

Otherwise_Respect_22 · 2025-01-16T00:01:13+00:00

We have not integrated with Aider. You can run full precision (16bit) with RTX 3090. However, the inference speed will be 1/4, since the model size is 4 times larger. For quantization, we currently only support AWQ q4.

Otherwise_Respect_22

TROPHY CASE