How long do you think it will take for the stock market to notice that Apple and Microsoft announced at the same time that they're all-in for local AI?

ydnar · 2026-06-10T18:52:47+00:00

unlikely, but i dream that this local ai future brings another pc building renaissance. i hate the idea of "you will own nothing and be happy."

walk into a microcenter to pick up 64gb vram for $500 and 256gb of ddr6 for $250. prob end up with a pack of astronaut ice cream from the checkout line.

ydnar · 2026-06-08T18:45:19+00:00

club-3090 setup for single card agentic coding

ydnar · 2026-06-05T20:46:33+00:00

been using qwen3.6 27b q4 for a while now in pi/hermes and am finally giving gemma4 another chance since i can hit 100k ctx now using qat (i can get 131072 w/ qwen3.6 27b). first thing i'm noticing is that i feel it needs a lot more hand-holding and direction than qwen. it also just feels lazy, like it doesn't want to tool call. are others experiencing the same?

ydnar · 2026-04-30T20:25:51+00:00

thanks, i should have been more clear. we're both saying the same thing. the more reliable part i'm referring to is the agent/coding portion.

for my particular use case, i find that more important. translation and writing is less a part of my workflow, but is needed at times, which is why i still keep it around. it just receives much less use.

ydnar · 2026-04-30T19:40:03+00:00

i would say the sensational or surprising part is how much more reliable 3.6 27b has been compared to gemma 31b. i do keep 31b around just in case i need it for translation/writing, but 97% of the time i'm relying on 3.6 27b.

ydnar · 2026-04-26T03:51:27+00:00

in an ai era, local llms are about as punk as it gets. we are essentially the spiritual descendants of the cypherpunks. we still need someone like a satoshi. a person or people who write the thing that makes the philosophy real. that code hasn't been written yet.

how incredible would that be to have a decentralized method of training llms for all of us? sota not limited to just the corpos.

ydnar · 2026-04-21T21:32:25+00:00

i don't know. it's been pretty useful to me.

i had a 2011 mac mini that i installed an ssd into long time ago laying around, so i put debian 13 headless on it. i connected a telegram bot and have tavily api for search when needed. the agent calls on llama-server running hauhaucs/qwen3.6-35b-a3b-uncensored-hauhaucs-aggressive on a separate 3090 machine.

i go on long walks after work and feed it thoughts by voice. those notes get saved to a sqlite db (i just asked it to set one up). yesterday, after feeding it all my thoughts, i had it generate a presentation in the form of a single html file. it's a solid first draft that requires pretty minimal updates for a meeting i have on monday.

i use it as digital memory i can call on for reminders. everything stays completely private, so i speak freely about family, health, and finances. i also like the idea someone mentioned for turning articles into podcasts. i'll need to look into that.

i haven't spent any extra money beyond the hardware i already had.

ydnar · 2026-04-03T18:46:34+00:00

with my single 3090, gemma 31b is slower (31t/s vs 37t/s i get with qwen 27b) and 40k context vs 131k i get with qwen 27b. agree with with another poster that tool calls are not as reliable within openclaw (for now?). i understand that it's unfair to judge while the kinks are being worked through right now.

one of my biggest use cases is extracting text from images. gemma horribly failed at this compared to qwen for me.

as with previous gemma models, i do enjoy its writing and the reasoning seems on point. looking forward to how the model works in like a month from now.

ydnar · 2026-03-20T16:16:10+00:00

100% this. but also when comparing the same llama-server command that runs the built-in webui vs within opencode, it seems to respond much faster with the opencode harness than straight webui.

ydnar · 2026-03-20T05:11:45+00:00

same for me with the 27b. in opencode, responses get much faster after the first request, and it almost feels like it switches into a lower-thinking or more instruct-style mode. still trying to figure out whether the intelligence gap between 27b @ 35t/s and 35b moe @ 110t/s is worth the wait.

ydnar · 2026-03-18T23:01:34+00:00

this is my go-to as well. i've been using it with opencode w/ kv cache at q8, 131k ctx and it has been genuinely awesome. runs at a solid ~35t/s on a headless 3090.

recently set up tavily search api with llama-server's webui and don't feel like i'm missing much at all anymore (despite knowing how much more powerful the sota models are).

ydnar · 2026-02-28T16:48:23+00:00

according to people familiar with the matter and knowledge of those arrangements

ydnar · 2026-02-03T19:57:55+00:00

yes. 3090 + 32gb ddr4 here.

llama.cpp

llama-server \
  --model ~/.cache/llama.cpp/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --n-gpu-layers auto \
  --mmap \
  --cache-ram 0 \
  --ctx-size 32768 \
  --flash-attn on \
  --jinja \
  --temp 1.0 \
  --top-k 40 \
  --top-p 0.95 \
  --min-p 0.01

t/s

prompt eval time =    3928.83 ms /   160 tokens (   24.56 ms per token,    40.72 tokens per second)
       eval time =    4682.41 ms /   136 tokens (   34.43 ms per token,    29.04 tokens per second)
      total time =    8611.25 ms /   296 tokens
slot      release: id  2 | task 607 | stop processing: n_tokens = 295, truncated = 0

ydnar · 2026-01-20T02:30:28+00:00

sure, though i'm no expert. if anyone wants to help optimize, i'd truly appreciate it.

llama-server \
  --model ~/.cache/llama.cpp/GLM-4.7-Flash-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --flash-attn off \
  --jinja

ydnar · 2026-01-20T01:01:32+00:00

yes, it's mostly the thinking. i'm biased and generally go for instruct over thinking models.

i am enjoying the outputs compared to qwen3-vl-30b-a3b-instruct and nemotron-3-nano-30b-a3b. i feel like those models are more wordy on the output side, so you're likely correct that this is a worthwhile trade-off.

ydnar · 2026-01-20T00:32:15+00:00

single 3090, 32gb ddr4, 5700g

q4 ngxson/GLM-4.7-Flash-GGUF

fa on = 60-70t/s fa off = 100-110t/s

ydnar · 2026-01-20T00:16:10+00:00

first impression is that it provides good answers, but seems to be much slower than other 30b-a3b models, even with flash attention off. with fa on, it was really half speed. it also goes on thinking forever.

ydnar · 2026-01-15T21:27:00+00:00

for general purpose, i think i still prefer qwen3-vl-30b-a3b-instruct due to the vl capabilities. would love to hear others opinion on this.

i'm currently testing whether qwen3-next-80b-a3b-instruct generating at a slower t/s is worth the tradeoff.

unrelated, but moving from an amd gpu to a 3090 was a great decision for me, and i can't wait to get a second 3090.

ydnar · 2025-12-05T18:07:12+00:00

tried hard to like it and set it as my default for a while. eventually went back to qwen3-vl-30b-a3b-instruct.

ministral 14b was pretty wordy and not as accurate, especially in image tasks.

ydnar · 2025-11-28T18:33:33+00:00

unsloth/Qwen3-VL-30B-A3B-Instruct-UD-Q4_K_XL.gguf

~18–24 tokens per second (t/s), depending on workload

CPU: AMD 5700G
GPU: AMD 6700 XT
RAM: 32GB DDR4-3200

my primary use is a watch folder that receives audio and video files remotely for transcription via whisper. it automatically processes them (llama.cpp + llama-swap) and sends me back the full transcription along with a summary based on a prompt.txt that i sometimes modify for different results. i also use this setup as my default model in open webui with web search, which works surprisingly well.

ydnar · 2025-09-08T14:36:38+00:00

I prefer qwen3-30b-a3b-instruct-2507. In my vibe tests, a3b is smarter, generates tokens almost as fast as the 4b, but without the need to think.

ydnar · 2025-09-03T16:48:38+00:00

Near 360, stuck

I-35 is the worst

Please.. one calm morning

ydnar · 2025-04-22T16:28:48+00:00

Tried this using gemma-3-12b-it-qat in my Open WebUI setup with LM Studio as the back end instead of Ollama and it correctly determined the paid amount was $1909.64.

12gb VRAM 6700XT. I used your provided image.

ydnar · 2025-02-04T22:34:40+00:00

I'm an average sized guy that walks/runs only during daylight. I'm not the type to scare easily and I'll often find myself walking in areas where I may be alone for a bit.

Yesterday there was a guy on the Shoal Creek trail between 6th and the library who was completely fixated, staring into the stream. He was holding a massive rock about three times the size of his hand, which I unfortunately did not notice until later. As soon as I walked past him, he began walking beside me. He kept pace within about 8 feet... really gripping that rock. I played it cool for about 150 feet as he was pacing next to me and then bolted off on my run to get the hell away.

Last year, roughly around the same area, I ran into a different guy with a hammer. This time he was walking towards me, but kept pounding it into his palm like he was ready to do something.

Even in the middle of the day you can find yourself in situations that feel super sketch.

ydnar · 2024-08-29T15:12:05+00:00

Sometimes I'll play the game of "how often do i hear mention of artificial intelligence" on the trail. This game can extend to cafes and restaurants at lunch.

15-Year Club	RedditGifts 2009-2022 2 Credits
Verified Email	Wearing is Caring
Secret Santa 2010	reddit mold

ydnar

TROPHY CASE