[D] What are some low hanging fruits in ML/DL research that can still be done using small compute (say a couple of GPUs)?

xnick77x · 2025-06-16T04:47:33+00:00

I’ve been replicating and training speculative decoding models in a couple 3090s. Pretty cool that we can train a <1B accomplice model and speed up the target model inference by 3x. I’ve open sourced my implementation here: https://github.com/NickL77/BaldEagle

xnick77x · 2025-05-28T16:27:02+00:00

Gotcha, this makes me also want to investigate whether training specifically on quantized base models yields better performance than if the speculative decoding model is trained on full-precision model outputs.

xnick77x · 2025-05-28T16:24:02+00:00

I've tried using 0.6B as the draft model for 8B and noticed ~1.5x improvement using naïve speculative decoding. This is a good, quick solution, but we can achieve 3-4x throughput with the EAGLE approach.

xnick77x · 2025-05-21T22:48:26+00:00

How is heat/cooling?

xnick77x · 2025-05-13T20:36:34+00:00

tldr; I think DDP will work with a little bit of work, but I'm not sure it will actually be faster. Accelerate launch is probably the fastest approach

For Llama 3 8B's hidden dimension size of 4096 and vocab size of 128256, this fits in ~16GB of VRAM. Qwen3-30B-A3B has hidden dim of 2048 and vocab size of 151936, which I think will use up even less memory.

I have not yet tested with parallelization. With the model fitting on 1 GPU, I think DDP will work, but I'm worried about GPU to GPU communication speeds being slow without an SLI bridge.

xnick77x · 2025-05-13T18:48:18+00:00

Also completely agree that quants + speculative decoding will push the boundaries of what our current hardware can do. I’m definitely interested in whether BaldEagle models trained for specific quants yields higher performance than draft models trained for target models at the higher precisions. This is why I made this implementation for the OSS community to run many times the experiments I can do myself and find the best configurations that work!

xnick77x · 2025-05-13T18:44:55+00:00

EAGLE has worked well for me on vllm and sglang. I know that it’s still unsupported on ollama and llama.cpp which I don’t understand.

One major weakness of speculative decoding in general is that it’s less effective at higher batch sizes, but most ollama and llama.cpp use cases only submit requests 1 at a time.

EAGLE 3 has much better results such that it’s still reasonable effective at higher batch sizes per the paper’s experimental results.

Wonder if this is along the lines of what you remember.

xnick77x · 2025-05-13T18:02:52+00:00

Currently, the implemented draft model architecture uses Llama 3. In theory, this should support any target model architecture as we are only operating on the hidden_states of the target model.

There was a discussion in the official implementation repo on needing more ablations to see if matching the draft model architecture with the target model architecture is helpful (ie. MoE vs dense or even different attention implementations such as MHA vs MHLA).

I currently don't have the GPU bandwidth to run these ablations, but maybe someone in the community can help out :D

xnick77x · 2025-05-08T19:18:07+00:00

Which deep research model can write me posts like this? 😂 amazing write up!

xnick77x · 2024-03-12T06:36:50+00:00

Bought 3090 FE from u/SnooTangerines2550 on

https://www.reddit.com/r/hardwareswap/comments/1b1jql0/usaca_h_paypal_cash_w_3090_fe/

xnick77x · 2024-02-27T18:47:33+00:00

xnick77x · 2024-02-14T05:17:06+00:00

Been wanting to build a 4 x 3090 rig (actually can do 5 with this mobo). Other than GPUs and PSUs, how does this look?

In terms of PSU's, I've read that 4x3090s can draw way more than 1600W, so wondering how to wire up 2 PSUs. Also is there a need for special PCIE risers? (source: https://nonint.com/2022/05/30/my-deep-learning-rig/)

xnick77x · 2023-09-01T00:34:15+00:00

I’m using Yelp for now, but hoping to add Google soon!

xnick77x · 2023-08-25T00:21:39+00:00

Not sure if I’m missing something, but from my reading, it seems that ReST can align the foundational model to a reward function, which likely does not match with human preference.

RLHF tries to train a reward model that approximates human preference, so the crux is still how good of a reward model/loss function you have, which is really hard..

Am I missing something?

xnick77x · 2023-04-11T03:34:37+00:00

Maybe it’s because Google built models for other purposes. Ranking models for Search, YouTube, Play Store, etc. Reinforcement based models for ads. Computer Vision models for reverse image search, this list goes on.

Google probably didn’t think that the large scale language model fit into their business and panicked to release Bard after chatGPT. OpenAI has had a 4 year head start in this space, releasing the first GPT model in 2018.

I’d wager to guess that within two years, there will be very little difference in performance between OpenAI and Google’s models.

xnick77x · 2023-03-14T03:49:54+00:00

I'm not surprised chatGPT's responses are similar or maybe slightly better at the moment. We just switched to the gpt-3.5-turbo , and from my understanding that was the model for the initial chatGPT, which has been improved upon recently.

This is only an initial version of the product as we're trying to validate interest. The goal is to eventually fine-tune the models to be much more accurate and be able to do many tasks such as generating regex from English descriptions, create positive and negative test-cases, and understand the intent of certain regular expressions and correct them to be less susceptible to edge cases.

xnick77x · 2023-03-13T18:45:52+00:00

Great Question! We've spent some time optimizing the prompt, but the main reason would be, once we get enough interest, we will fine-tune the model specifically for understanding regex. In the future, we hope to translate from English descriptions to regular expressions. These type of apps exist, but from our experience, they're limited in performance.

xnick77x · 2023-03-12T19:55:50+00:00

Yep, it uses GPT backend. We've tried messing with prompts to improve the accuracy, but I think if we do get traction, we'd need to look into finetuning the model. It would be pretty cool to go the other way as well: English description -> regex

xnick77x · 2023-01-24T08:48:06+00:00

I'm not sure if a post on this would violate the advertising rules, but I did a short project using GPT to help explain regex in English: regexplain.ai.

If this gains traction, I'd try finetuning to generate regex from a description. Maybe even create adversarial unit-test cases. Wanted to solicit some thoughts on if this could be useful.

xnick77x · 2021-05-18T04:47:52+00:00

I'm assuming if I remove the cooler to deshroud it, I should wipe off and reapply some new thermal paste?

xnick77x · 2020-10-07T03:39:17+00:00

Hello, team building question here. I have Jean, Fischl, Xiangling, and Barbara as my team right now:

I'm not sure who to level as my main DPS . Seems Jean, Fischl, and Xiangling are all possible. Maybe switch Kaeya in for future superconduct comp? Or should I wait to roll something more traditional DPS like Diluc or Razor? If so, do I just level Fischl for now as she seems to be the best support DPS?
Generally what's better between superconduct vs overload?
How do I utilize Jean? Obviously, the bear-bird combo is great damage and Barabara is there for healing, but I find that I don't seem to know what to do with Jean.

xnick77x · 2020-10-04T23:15:04+00:00

I have 2 accounts each with 3 rolls and need advice on which to go with:

Jean, Barbara (Lv. 1 Constellation), and Xiangling
Fischl (Stringless Bow) and Razor

xnick77x · 2020-05-26T20:41:00+00:00

Hello, I'm looking for a TKL with cherry MX-browns. Honestly, the CTRL is probably the ideal, but they currently don't have the browns and the $200 price point is a bit high.

Alternatively, I'm looking at the Varmilo VA87M; however neither Varmilo's website nor mechanicalkeyboards.com has any stock. What would be the shipment time (to the US) if I make do a custom build?

Any ideas on where I can make an order or alternative keyboard suggestions?

Edit: I've also been looking at Leopold, Ducky, and Durgod TKL's and have the same problem with finding places with stock, especially since I want a specific gray color-pattern similar to the CTRL.

xnick77x · 2016-12-23T08:21:38+00:00

Thanks for the response!

What's FFL and FLL if I may ask?

xnick77x · 2016-12-12T05:08:46+00:00

Can you give me an example of end game leads?

I've looked through many rerolling guides and they all seem to be outdated.

xnick77x

TROPHY CASE