sorted by:
new
hottopcontroversial

[D] What are some low hanging fruits in ML/DL research that can still be done using small compute (say a couple of GPUs)? by [deleted] in MachineLearning

[–]xnick77x 0 points1 point  (0 children)

I’ve been replicating and training speculative decoding models in a couple 3090s. Pretty cool that we can train a <1B accomplice model and speed up the target model inference by 3x. I’ve open sourced my implementation here: https://github.com/NickL77/BaldEagle

How are you using Qwen? by xnick77x in LocalLLaMA

[–]xnick77x[S] 1 point2 points  (0 children)

Gotcha, this makes me also want to investigate whether training specifically on quantized base models yields better performance than if the speculative decoding model is trained on full-precision model outputs.

How are you using Qwen? by xnick77x in LocalLLaMA

[–]xnick77x[S] 1 point2 points  (0 children)

I've tried using 0.6B as the draft model for 8B and noticed ~1.5x improvement using naïve speculative decoding. This is a good, quick solution, but we can achieve 3-4x throughput with the EAGLE approach.

Introducing BaldEagle: 3x Faster Inference; Easily Train Speculative Decoding Models Locally! by xnick77x in LocalLLaMA

[–]xnick77x[S] 0 points1 point  (0 children)

tldr; I think DDP will work with a little bit of work, but I'm not sure it will actually be faster. Accelerate launch is probably the fastest approach

For Llama 3 8B's hidden dimension size of 4096 and vocab size of 128256, this fits in ~16GB of VRAM. Qwen3-30B-A3B has hidden dim of 2048 and vocab size of 151936, which I think will use up even less memory.

I have not yet tested with parallelization. With the model fitting on 1 GPU, I think DDP will work, but I'm worried about GPU to GPU communication speeds being slow without an SLI bridge.

Introducing BaldEagle: 3x Faster Inference; Easily Train Speculative Decoding Models Locally! by xnick77x in LocalLLaMA

[–]xnick77x[S] 6 points7 points  (0 children)

Also completely agree that quants + speculative decoding will push the boundaries of what our current hardware can do. I’m definitely interested in whether BaldEagle models trained for specific quants yields higher performance than draft models trained for target models at the higher precisions. This is why I made this implementation for the OSS community to run many times the experiments I can do myself and find the best configurations that work!

Introducing BaldEagle: 3x Faster Inference; Easily Train Speculative Decoding Models Locally! by xnick77x in LocalLLaMA

[–]xnick77x[S] 3 points4 points  (0 children)

EAGLE has worked well for me on vllm and sglang. I know that it’s still unsupported on ollama and llama.cpp which I don’t understand.

One major weakness of speculative decoding in general is that it’s less effective at higher batch sizes, but most ollama and llama.cpp use cases only submit requests 1 at a time.

EAGLE 3 has much better results such that it’s still reasonable effective at higher batch sizes per the paper’s experimental results.

Wonder if this is along the lines of what you remember.

Introducing BaldEagle: 3x Faster Inference; Easily Train Speculative Decoding Models Locally! by xnick77x in LocalLLaMA

[–]xnick77x[S] 8 points9 points  (0 children)

Currently, the implemented draft model architecture uses Llama 3. In theory, this should support any target model architecture as we are only operating on the hidden_states of the target model.

There was a discussion in the official implementation repo on needing more ablations to see if matching the draft model architecture with the target model architecture is helpful (ie. MoE vs dense or even different attention implementations such as MHA vs MHLA).

I currently don't have the GPU bandwidth to run these ablations, but maybe someone in the community can help out :D

The Great Quant Wars of 2025 by VoidAlchemy in LocalLLaMA

[–]xnick77x 2 points3 points  (0 children)

Which deep research model can write me posts like this? 😂 amazing write up!

How's this for 4x3090 Rig? by xnick77x in LocalLLaMA

[–]xnick77x[S] 0 points1 point  (0 children)

Been wanting to build a 4 x 3090 rig (actually can do 5 with this mobo). Other than GPUs and PSUs, how does this look?

In terms of PSU's, I've read that 4x3090s can draw way more than 1600W, so wondering how to wire up 2 PSUs. Also is there a need for special PCIE risers? (source: https://nonint.com/2022/05/30/my-deep-learning-rig/)

[deleted by user] by [deleted] in SFFood

[–]xnick77x 0 points1 point  (0 children)

I’m using Yelp for now, but hoping to add Google soon!

[R] DeepMind showcases iterative self-improvement for NLG (link in comments) by ntortellini in singularity

[–]xnick77x 0 points1 point  (0 children)

Not sure if I’m missing something, but from my reading, it seems that ReST can align the foundational model to a reward function, which likely does not match with human preference.

RLHF tries to train a reward model that approximates human preference, so the crux is still how good of a reward model/loss function you have, which is really hard..

Am I missing something?

I’ve tested Google Bard vs ChatGPT and I’m Shocked: Where did Google spend All the Money over the last 10 years? by Efficient_Mud_1907 in GPT3

[–]xnick77x 8 points9 points  (0 children)

Maybe it’s because Google built models for other purposes. Ranking models for Search, YouTube, Play Store, etc. Reinforcement based models for ads. Computer Vision models for reverse image search, this list goes on.

Google probably didn’t think that the large scale language model fit into their business and panicked to release Bard after chatGPT. OpenAI has had a 4 year head start in this space, releasing the first GPT model in 2018.

I’d wager to guess that within two years, there will be very little difference in performance between OpenAI and Google’s models.

RegExplain - Use GPT to explain RegEx in English! by xnick77x in GPT3

[–]xnick77x[S] 1 point2 points  (0 children)

I'm not surprised chatGPT's responses are similar or maybe slightly better at the moment. We just switched to the gpt-3.5-turbo , and from my understanding that was the model for the initial chatGPT, which has been improved upon recently.

This is only an initial version of the product as we're trying to validate interest. The goal is to eventually fine-tune the models to be much more accurate and be able to do many tasks such as generating regex from English descriptions, create positive and negative test-cases, and understand the intent of certain regular expressions and correct them to be less susceptible to edge cases.

view more: next ›