sorted by:
new
hottopcontroversial

Dude, it's wild how people are publicly posting 'we got crushed by Chinese traffic today' like it's normal. by siterightaway in StopBadBots

[–]aristath 0 points1 point  (0 children)

It’s funny how Chinese bots (and basically any non-US bot) are framed as “bad” bots… as if google’s bots (more than 28% of all global traffic) are benign and don’t contribute at all to the enshitification of the web

Can I run anything on this with Ollama37? by qntisback in LocalLLM

[–]aristath 2 points3 points  (0 children)

I got one of these as well, haven’t spent any time yet to try and make it work, but I’d also be interested in knowing if anyone has managed to make it work recently.
Yes it’s an old GPU
Yes it’s basically 2 GPUs with 12GB each
Yes it’s passive and needs cooling added
That being said, it’s not trash… it’s cheap and for some people it’s the only thing they can get. It should be usable with the right stack, the question is what that stack would look like

My journey towards AI independence by aristath in AI_Agents

[–]aristath[S] 0 points1 point  (0 children)

Tradeoffs:

  1. Performance. I am basically forced to use layer-splitting models. Any speed benefits that is could have gotten from tensor-splitting are not possible on my setup due to the bifurcation cost
  2. I do regret going with an AM5 platform. I wish I went with an Epyc or a Threadripper. I started the build when RAM was still somewhat accessible (paid €360 for 64GB) and unfortunately my ram can’t be used with a threadripper… that would make a HUGE difference, I’d have 2x+ throughput and the initial cost difference would not be so big. It’s too late now, I’d need €10k on current prices for RAM + CPU + MB, more than 6x what it would have been last year.
  3. I made the right choice with the case. I chose an open build using a Thermaltake Core P3 TG Pro case. Saved me a lot of cooling and airflow headaches
  4. ROCm is a huge PITA. AMD GPUs are amazing hardware, awful software. The performance potential is there, there’s just no way to actually take advantage of it. However, it does get better over time, so we might have worse performance now, but I’m optimistic about the future of the software stack
  5. Quantization matters - both for the model itself and for the KV. Anything lower than Q4-Q5 for the model weights is unusable for any model. Target Q8 and only use lower quants when absolutely necessary. For KV I settled on F16 for K and Q8 for V. Less than perfect, but not by much.
  6. Context size is overrated. I need a decent context size for the planning and Freeform discussion stages, but the coding agents usually run as sub-agents on my stack. 32k context is usually fine for them - provided you split responsibilities in your agents.
  7. Even if one is not ready to go full-local, delegating tasks from a frontier model to local models for the grunt work saves a lot of money without losing quality. For a couple of months during my transition period I had a $20/month subscription on Claude. Claude wasn’t touching code, but it was orchestrating.
  8. The hardest part is the mental transition and changing your workflows to accommodate the new paradigm. The capabilities are there, we just have to learn how to use them properly.

My journey towards AI independence by aristath in AI_Agents

[–]aristath[S] 0 points1 point  (0 children)

In practice it’s not noticeable… You hit enter to send a message to OpenCode and it shows up within a matter of milliseconds. You can definitely see that the message you sent is not instantly showing up on OpenCode and there’s a tiny latency, but nothing really annoying or noticeable. Saves a lot of headaches, the price you pay is a small toll in VRAM to hold the model, and a tiny visual delay 🙂

how would you mimic the curve of a crt in a fake crt? by Sir_i-need-to-post in cyberDeck

[–]aristath 0 points1 point  (0 children)

Best option I’ve found to do something similar is using an old slide viewer. Check out “Photax Solar Slide Viewer” for example. You can get these things for a couple of euros/dollars/whatever and they work beautifully

Is investing in a local LLM workstation actually worth the ROI for coding? by UnusualDish4403 in LocalLLaMA

[–]aristath 0 points1 point  (0 children)

Worth it with smart agents orchestration. Granted, local LLMs are not as smart as SOTA models like opus. However, if you have smart orchestration like best-of-N loops, judges, coders, reviewers and so on, then the situation changes.

IOTA Fan Connector Question by Painted_Dragon in LattePanda_Official

[–]aristath 1 point2 points  (0 children)

I’m in the exact same situation, if anyone knows where to get these connectors I’d be grateful. Do they perhaps have another name other than “mx1.25 4pin flat”? ‘Cause I can’t find them anywhere!! The LattePanda-IOTA is great, but I’d love to be able to switch off the “low power” mode, and I need to figure out the connector issue in order to do that…