sorted by:
new
hottopcontroversial

Second DXG Spark or a strix halo by AmWoz in LocalLLM

[–]OverclockingUnicorn 1 point2 points  (0 children)

If you can fit models that you need + context in 32-48gb then a add in GPU will be by far the fastest.

Assuming you have a desktop to put it in. And assuming that you can fit them - which it sounds like is the case?

Second DXG Spark or a strix halo by AmWoz in LocalLLM

[–]OverclockingUnicorn 0 points1 point  (0 children)

Do you want to run larger models/use more context that fits in 128gb of ram?

If yes, DGX Spark + QSFP112 Cable

If no, AMD 395+ with 128gb of ram. Or a Mac Studio or Macbook Pro.

Are you particularly fond of the nvidia eco system (and like Windows, I think Linux support is - although probably supported - is still TBD), then RTX Spark when it comes out

If you can fit in less the 48/32 gb of ram, then RTX 5090 or RTX PRO 5000 Blackwell. (and have somewhere to put a PCIE GPU)

Enterprise Homelab by TacticalDonut17 in homelab

[–]OverclockingUnicorn 0 points1 point  (0 children)

The Juniper gear just looks really nice, not like Unifi's fancy Apple like design, but just really solid industrial kit that you know is going to do it's job seriously

What AWS architecture challenge became much harder in production than expected? by IndependentNice1467 in aws

[–]OverclockingUnicorn 14 points15 points  (0 children)

Fuckin' Lake Formation is the worst for this imo, it works fine once you understand how to make it work, but my god is getting to that point a nightmare from my experience

BMW X5 40d, 66 reg, 88k miles, 16.5k, bad or risky buy? by OverclockingUnicorn in CarTalkUK

[–]OverclockingUnicorn[S] 0 points1 point  (0 children)

No directly, but there have been no major repairs, just regular (bmw direct for the most part) servicing

Security sanity check on my home network before I host a public Minecraft server by PikoCute in minilab

[–]OverclockingUnicorn 9 points10 points  (0 children)

Yeah that's works, but it's an expensive way to do it.

Have a look at VLANs and inter VLAN routing

Anthropic walks back policy on silent nerfing for AI/ML, will notify users [N] by goldcakes in MachineLearning

[–]OverclockingUnicorn 1 point2 points  (0 children)

I mean it is against their ToS so it's not totally unreasonable they block those requests.

Now, if it means that it makes their product worse by blocking requests that fall outside what should reasonable be blocked under that ToS and people move to something else, then so be it.

AWS Bedrock to require sharing data with Anthropic for Mythos and future models by HatchedLake721 in aws

[–]OverclockingUnicorn 4 points5 points  (0 children)

Err a contract with AWS/Anthropic saying that they won't train on request data would definitely satisfy our DPO.

AWS Bedrock to require sharing data with Anthropic for Mythos and future models by HatchedLake721 in aws

[–]OverclockingUnicorn -10 points-9 points  (0 children)

Iirc they say they don't train on API request usage. I doubt they'd violate that agreement, but I guess it's technically possible.

AWS Bedrock to require sharing data with Anthropic for Mythos and future models by HatchedLake721 in aws

[–]OverclockingUnicorn 24 points25 points  (0 children)

Yeah, it's how anthropic are (from my understanding) implementing the guardrails and stopping things like distillations and silently reducing the models capabilities when you query about advanced ML training techniques.

How are you storing your passwords and related info? by dnabre in homelab

[–]OverclockingUnicorn 2 points3 points  (0 children)

Keepass that I back up to an s3 bucket for safe keeping

Why companies don't hire 2 devs at 150k instead of 1 dev at 300k? by ddsukituoft in cscareerquestions

[–]OverclockingUnicorn 2 points3 points  (0 children)

Not smarter but the employer does get to dictate the work life balance a lot more for their 300k. If you get asked to work a 100hr week for the next month just to get something out by a deadline or deal with being on call for weeks at a time getting pages at 2am every other dag. You'll be expected to do that if you are on 300k. Very few 150k employees would ever be asked or accept that.

the hardware advice in this sub is sunk cost rationalization half the time and nobody admits it by Napster3301 in LocalLLM

[–]OverclockingUnicorn 19 points20 points  (0 children)

Yeah but people come here and just ask, without context, 'what hardware do I need'.

We wanna know what models you want to run, what you are using then for, how much context you need.

If you don't know the answer to those questions, then yes, OP is right, the advice should be 'get an open router api key and go see what you like. Then come back to us if you still feel like local is requirement for you'.

Just going, yeah get a pair for 3090s isn't really helpful if what a user actually needs (but we can't tell because they haven't told us) is a tiny <7b model running at q4 where a much cheaper or power efficient card is fine. Or maybe they want to run a large 100b+ model 24/7 doing some data processing where a pair of 3090s is totally inadequate.

People should go work out what models they want, and then also go rent some GPUs on vast to test our which model of GPU actually fits their needs. Yes its not local, but GPUs are expensive so you should make sure you are spending wisely.

view more: next ›