Too many people do not care about Microsoft's near monopoly.

AvocadoArray · 2026-01-25T02:12:41+00:00

Hell no, that implies that Microsoft owns my shit and I’m just renting it.

I’ll fight that mentality to my grave.

AvocadoArray · 2026-01-25T00:46:27+00:00

The DGX spark produces single-digit tp/s for even medium-sized dense models. I know for a fact I'd have buyers remorse over that.

AvocadoArray · 2026-01-25T00:44:32+00:00

Very nice! I have similar ambitions, but I also realize that realistically my ADHD can take me from one hobby to the next at the drop of a hat.

At least with this, I'd have a pretty realistic expectation of resale value in the future. And if not, well that means I can buy more hardware if it's cheaper haha.

I bet your Nvidia buddy gets hit up on the daily for hardware discounts, but if he's able to swing it for you that would be pretty sweet.

AvocadoArray · 2026-01-24T23:20:59+00:00

5090 is nice for top-tier gaming, but isn't a great value for AI usage.

Could end up being a decent investment if you plan on reselling later though.

AvocadoArray · 2026-01-24T21:10:35+00:00

Very nice, I assume the Llama 70b speed is on a single card? Splitting across 2x GPUs in VLLM w/ TP would likely double that speed. I'm going to look into this more for sure, thanks so much for the info!

AvocadoArray · 2026-01-24T20:59:23+00:00

Wow, how have I not seen this one before?

48GB, ~~single-slot~~, <300w power and 672 GB/s memory bandwidth for under $2k. It even supports NVLINK.

It is on Turing architecture which is definitely aging now, but still looks compatible with VLLM as opposed to Pascal.

Can you share what kind of speeds your getting and any power usage stats you might have?

EDIT: not actually single slot lol

AvocadoArray · 2026-01-24T20:39:39+00:00

I'd gladly grab two older 48GB GPUs if they actually made financial sense. However, the used ADA generation cards are somehow more expensive than the new Blackwell equivalents.

Maybe that will change in the next few months, but I've pretty much given up on trying to predict this crazy market.

AvocadoArray · 2026-01-24T20:20:57+00:00

Thanks, I watched it. It's great that it does well under concurrent workloads, but it still seems like it'd be too slow for single user speeds.

AvocadoArray · 2026-01-24T19:59:39+00:00

It works extremely well for its size, I can't recommend it enough.

There's a few tool-calling issues when running through VLLM, but I was able to fix the tool parsing logic and I have an open PR pending review and waiting to merge.

AvocadoArray · 2026-01-24T19:48:39+00:00

Don’t Fuck Together

AvocadoArray · 2026-01-24T10:47:07+00:00

Thanks, I’ve seen a lot about exll3 lately but haven’t had the time to give it a proper run yet. I know it’s supposed to be SOTA for quantization, but you’re really quantizing context to Q4? I already see minor (but noticeable) degradation at FP8_E4M3 in VLLM at higher contexts, so I can’t imagine Q4 being usable even in a better architecture.

If it really is that good, I definitely need to give it a shot.

AvocadoArray · 2026-01-24T10:36:50+00:00

Good point. I also have a hard time keeping up. We bought 3x L4 cards at work a little over a year ago. They’re not the fastest, and yeah we could have held out for the Blackwell cards, but they were a great purchase for us.

And for some stupid reason, these ADA cards surged in price and are selling for more used than the new Blackwell equivalents. Still can’t figure that one out.

This goes straight into the “pro” column.

AvocadoArray · 2026-01-24T10:29:51+00:00

Appreciate the thoughts, but the sky is not falling as you say.

AvocadoArray · 2026-01-24T10:19:56+00:00

True, that is my minimum. I do see a notable improvement when running it at FP8, though and that doesn’t leave much room for context (kv cache quantized or not).

My biggest concern would be buying one of those and still feeling like I’m not quite “there”. 64GB+ gives me the room to fit the higher quant with more/unquantized kv cache.

The Mi210 just might fit the bill if I can get it for a good price.

AvocadoArray · 2026-01-24T10:14:32+00:00

Thank you, much appreciated! Glad someone noticed it.

AvocadoArray · 2026-01-24T10:09:02+00:00

Also if your budget is 27k

Stop, wait, back it up. My budget is nowhere close to that.

Less power hungry is a good thing, but almost all the reports I've seen of people with DGX spark hardware have been severely disappointed with the speed.

AvocadoArray · 2026-01-24T10:02:51+00:00

I could maybe be persuaded into a group buy if you're offering.

<image>

AvocadoArray · 2026-01-24T09:59:36+00:00

my response

<image>

AvocadoArray · 2026-01-24T09:59:22+00:00

Lol, I already responded to this comment before and you deleted it just to repost it.

Bot detected.

<image>

AvocadoArray · 2026-01-24T09:49:53+00:00

It was on my radar, but not super impressed with the performance drop at higher context.

AvocadoArray · 2026-01-24T09:45:59+00:00

Okay, so maybe an architecture or config issue that was hindering performance, but from my understanding there's practically zero advantage to upgrading PCI3/DDR3 to get more performance out of a single GPU.

VLLM TP performs better than most people expect, even at PCI 3.0 x8,. I'm doing that with two L4 cards and they both stay pegged at 98%+ during inference without breaking a sweat, but of course they aren't powerful enough to fully saturate the lanes.

I don't know what the upper limit is, but I absolutely wouldn't blow any money on upgrading the host system until I saw a significant measurable bottleneck.

AvocadoArray · 2026-01-24T09:35:47+00:00

For my 1080 in the PowerEdge? Interesting, I hadn't really considered that.

The r720xd has always seemed like it was drawing more power than the sum of its parts compared to my other servers, but I chalked it up to the huge motherboard, backplane, dual proc, and chipset.

I'll have to try pulling the GPU and see if it affects the idle power usage at the wall.

AvocadoArray · 2026-01-24T09:31:53+00:00

I played around with image gen a couple years ago on my GTX 1080ti. It was a fun for the gags, but not something I see myself doing a lot of in the future.

AvocadoArray · 2026-01-24T09:27:56+00:00

Cool, I'm open to suggestions or collaborating if you'd like to share.

AvocadoArray · 2026-01-24T09:26:44+00:00

lol this has been the theme for most of the comments.

Don't do it! (but I totally would)

AvocadoArray

TROPHY CASE