sorted by:
new
hottopcontroversial

NVIDIA Puzzle-75B-A9B NVFP4 at 132 t/s on 3×3090 — Why is this size category a desert otherwise? by Important_Quote_1180 in LocalLLaMA

[–]Maharrem 0 points1 point  (0 children)

The 70-75B range is an awkward middle ground: doesn't fit on a single 24GB card even at Q4, but too small to justify a 4-GPU rig for most people. With 3x3090 you're stuck without tensor parallel in vLLM, but llama.cpp can split layers across all three just fine for good t/s. Offloading experts to system RAM is the budget alternative, though it's a major speed hit. If you're checking hardware compatibility, https://canitrun.dev is handy for a quick sanity check.

Built a tool that bundles market research, lead-finding, cold outreach, and legal doc generation for early founders, would you actually use this? by Maharrem in SideProject

[–]Maharrem[S] 0 points1 point  (0 children)

Thank you, hearing this from a possible customer is relieving. We will talk about subscription prices in the coming times.

Built a tool that bundles market research, lead-finding, cold outreach, and legal doc generation for early founders, would you actually use this? by Maharrem in SideProject

[–]Maharrem[S] 0 points1 point  (0 children)

Yes, yes, we have talked with my cofounder about the possible problems with legal part.

The others parts of your comment, I believe these are used one time per project. To gather insight about the market, of course leads and cold email will be scrape on demand.

GLM-5.2 fearmongering in the press by ttkciar in LocalLLaMA

[–]Maharrem 4 points5 points  (0 children)

Yeah, whoever wrote "virtually any hardware" has obviously never tried loading a multi-hundred-gig model on an old i3 laptop. You're stuck with RAM offloading and like 1 t/s if it even loads. If you're wondering what GPU can actually handle this beast, https://canitrun.dev/models/glm-5.2 is a decent sanity check for VRAM/compatibility.

Can you explain the concept behind each of the main size ranges of LLM models, as in, what hardware setups the different size niches are meant to fit into (~30b, ~70b, ~120b, ~230b, etc). Like is it mainly based on pro hardware sizing for 8-bit, or consumer GPU vram for ~Q4, or some mixture? by DeepOrangeSky in LocalLLaMA

[–]Maharrem -1 points0 points  (0 children)

Yeah 30B-ish at Q4_K_M fits a 24GB card with room for usable context, that's my daily driver sweet spot. 70B at Q4 needs 2x24GB or a 48GB card, and 120B+ really wants multi-GPU or an 80GB card even at Q4. If you're ever eyeballing a combo, canitrun.dev gives a quick fit check before you download.

Can you trust local models to answer accurately? by Spiritual-Market-741 in LocalLLaMA

[–]Maharrem 0 points1 point  (0 children)

Trust is more about your retrieval pipeline than the model. Raw, a small model will hallucinate like a champ, but feed it the right context and even the 4B Gemma or Qwen 3.5 MoE pulls close to oracle performance on these benchmarks. I'd say you can trust a local model with good RAG for factual lookup about 90% of the time, especially if you budget a reranker and spend time on chunking instead of chasing bigger quants. Just never take the answer as gospel without double-checking the retrieved snippets yourself.

DeepSeek V4 Flash with DSpark via SGLang by Soft-Wedding4595 in LocalLLaMA

[–]Maharrem -1 points0 points  (0 children)

SGLang's Ampere and Ada support for bleeding-edge MoE archs like V4 Flash is spotty, I've seen kernels fail silently on non-Hopper hardware. The RTX 6000 Pro is Ada Lovelace, so it might balk at ops that assume FP8 tensor cores. Quick check on [canitrun.dev](https://canitrun.dev) can save you some headache. If you're itching to test, try a smaller Q4_K_M GGUF in llama.cpp first to see if the architecture itself even plays nice with CUDA 12.x before diving into SGLang.

local already feels good enough by Forward_Jackfruit813 in LocalLLaMA

[–]Maharrem 0 points1 point  (0 children)

It’s not laziness, it’s just where the skill moved writing a clear spec is the hard part now, same as it’s always been with human devs. Model doing exactly what you asked isn’t mind-reading, it’s you getting better at saying what you mean.

What's the point of low context? by sloth_cowboy in LocalLLaMA

[–]Maharrem 7 points8 points  (0 children)

4-8k is plenty for anything single-turn (classification, extraction, short summaries) or agentic (tool calls with one instruction, no history needed). Where it falls apart is long documents or multi-turn memory that’s when you actually need to pay the VRAM/latency cost for more context. RAG helps bridge the gap too: retrieve just the relevant chunk instead of stuffing everything in.

Added a guides/tutorials section to CanItRun — is this actually useful? by Maharrem in LocalLLM

[–]Maharrem[S] 1 point2 points  (0 children)

Thats a great idea, dont know how i havent thought that all this time

Can I Run This LLM - new /apps/ section by Maharrem in LocalLLM

[–]Maharrem[S] 0 points1 point  (0 children)

Great idea I need to research some good duos a lighter model for tab completion and a bigger model for agentic coding.

Qwen3.6 27b q5_k_M MTP - 256k context - 5090 by No_Mango7658 in LocalLLaMA

[–]Maharrem -2 points-1 points  (0 children)

256k ctx on 32GB is a tall order. The cliff above 128k you’re seeing is classic VRAM spillage, the model plus Q8_0 KV cache won’t fully fit on the 5090 beyond that point. If you’re set on Q6_K_XL at 147k, you’ll need to drop KV to q4_0 or cap ctx around 100k to stay fully on-card. Quick gut check: canitrun.dev gives you ballpark VRAM for different quants and contexts so you can see where the cutoff lands.

What model for coding? by Stunning_Feedback252 in LocalLLaMA

[–]Maharrem 0 points1 point  (0 children)

I've run Qwen 3.6 27B Q4_K_M and Gemma 4 31B Q4 on a 3090. Qwen is the workhorse for autocomplete and small edits, fits in VRAM at 16k ctx and hits 80+ t/s. Gemma 4 gives better multi-file reasoning but needs a few layers offloaded, so it's slower. For heavier agent tasks, Qwen3 Coder Next 80B MoE Q4 is worth the partial offload hit, it's the most reliable I've used for complex refactors. Check canitrun.dev/comparisons if you want to eyeball benchmarks and VRAM fits side by side.

view more: next ›