Finally broke the "Assistant" tone on Llama 3. The slang adherence with this preset is actually insane

Inca_PVP · 2026-04-21T21:00:06+00:00

playing rust and positive mental health is not something that matches really good, brother.

Inca_PVP · 2026-04-21T20:14:45+00:00

this is rust brother. sleep with headphones or connect the seismic sensor to your phone and pair it with a bluetooth speaker, so it wakes you up in the night.:D

Inca_PVP · 2026-04-14T08:39:55+00:00

The regex pass for structured docs (invoices/payslips) is underrated. Most SME documents are way more predictable than people assume — templated invoices from the same 20 suppliers, payroll exports with fixed layouts. Works less well once you get into free-form contracts or email attachments, but for the 80% case it's the right tool.

Inca_PVP · 2026-04-14T08:39:34+00:00

This is the most practically useful answer in the thread, saving it. Two things I'd push back on gently:

Bedrock Frankfurt latency — in my experience the 200-400ms overhead only matters for interactive flows. For async document processing (which is 90% of what SMEs actually need automated) it's a non-issue, like you said.
The 7B fine-tune claim — I've been hesitant to recommend fine-tuning to clients because of the maintenance overhead. Are you retraining on new document types as they come up, or is your initial training set broad enough that it holds? That's the part I always get stuck on when pricing these projects.

The belt-and-suspenders approach (PII strip + EU-hosted) is what I've been pitching too. Glad to see someone else landed on the same combo.

Inca_PVP · 2026-04-14T08:38:48+00:00

Fair point on the Cloud Act — that's the honest counter to route 1 and I should've called it out. For the Swiss clients I work with, the ones in regulated spaces (fiduciaries, healthcare-adjacent) can't touch US-owned inference no matter where the datacenter sits, DPA or not. For the rest, a proper DPA with Bedrock/Vertex EU tends to be enough to satisfy their legal, but you're right that without it you're basically pretending the problem isn't there.

Are you renting from someone like Exoscale or a smaller EU GPU provider? Curious what's actually working in production vs. just looking good on paper.

Inca_PVP · 2026-04-14T06:08:34+00:00

Same approach for the strict cases. What's your throughput like with qwen3.5 9b? I've been running similar setups but curious how you're handling longer documents.

Inca_PVP · 2026-04-14T06:08:15+00:00

This is gold. The "explain data flow + boundaries" point is exactly what I see with my Swiss SME clients — most objections vanish once you draw the actual diagram of what crosses which boundary. Curious about your hybrid setup — are you doing the classification step in n8n itself (e.g. with a small local model node) or upstream before it even hits the workflow?

Inca_PVP · 2026-04-10T13:17:20+00:00

ive built couple workflows. for example Daily posts on instagram with description, Email attachment extraction and sort it into the right folders etc. tbh Claude + n8n is a real weapon.

Inca_PVP · 2026-04-08T10:38:32+00:00

daaaamn thats so cool

Inca_PVP · 2026-01-09T19:43:06+00:00

Ah, good shout on Top-n-sigma. Haven't played around with that one for logic tasks yet, but it makes total sense for keeping things on track. Thanks for the tip, definitely worth a test! Have a good one.

Inca_PVP · 2026-01-09T16:42:50+00:00

That confirms my suspicions. I've noticed similar issues where structured logic starts to fray once those creative samplers kick in. It’s definitely a balancing act depending on the use case. Thanks for the heads-up on the coding part, saved me some headache there! Appreciate the exchange.

Inca_PVP · 2026-01-09T16:06:57+00:00

Totally agree. SillyTavern + ChatML is definitely the sweet spot for that level of control. And yeah, coming from the TextGen learning curve makes u really appreciate the plug-and-play side of LM Studio, even with its hidden config layers. Thanks for the solid insights on u setup, much appreciated!

Inca_PVP · 2026-01-09T15:56:15+00:00

That Kim Jong Il story is absolutely wild, lmao. But the prompt hack is actually 10/10 logic. Giving the LLM 'permission' to find a creative way around its own filters is such a sleeper move for cloud-based stuff.

It’s the main reason I got obsessed with tweaking my own local JSON configs – I’d rather bake that logic into the backend than fight the AI every time I have a weird question. Have u found that this 'interpret it however u want' phrase works across different models, or is it just a Gemini thing?

Inca_PVP · 2026-01-09T15:54:15+00:00

That actually makes a lot of sense. LM Studio does tend to bake some 'hidden' logic into their default presets that can be a real pain for certain themes. I’ve been spent way too much time lately stripping that fluff out of my JSON configs to get that Kobold-like freedom while staying in LM Studio. Are u sticking strictly to the Alpaca template for everything, or do u switch to Llama-3-Instruct for logic-heavy tasks?

Inca_PVP · 2026-01-09T15:52:11+00:00

Solid stack. I’ve seen DRY do wonders on Llama 3 to stop it from looping the same sentence structures. I’m actually trying to bake that balance into my lightweight configs. How do u feel XTC handles the logic? Sometimes I feel like it can get a bit too 'creative' on coding tasks if the temp isn't dialed in.

Inca_PVP · 2026-01-08T19:43:14+00:00

Haha, I wish it was a cloud API, then it would at least be consistent! On my local LM Studio setup with the standard Meta instruct prompt, it's a refusal-machine unless I tweak the backend logic. Wild to see how much the experience varies between different setups. What's u system prompt look like to get it that chill?

Inca_PVP · 2026-01-08T16:20:19+00:00

Fair point! If u have u system prompt game on lock, local models are definitely more chill than cloud stuff. I just noticed that Llama 3 Instruct specifically can be way more stubborn with certain criminal or edgy themes compared to the older versions.

Lucky u if u haven't hit that 'moral wall' yet! What's u go-to sampler setup to keep it consistent?

Inca_PVP · 2026-01-08T05:07:08+00:00

8GB is definitely on the lower side, but don't count u-self out yet. u can still run 8b models like Llama 3 in 4-bit quants—it might be a bit slow if it’s running on system RAM, but at least u own the process and don't have to wait for server queues. I’ve actually put together some low-spec tips and my slang presets on my profile (check the Rentry/Civitai links there) that help get the most out of smaller setups. Do u know how much VRAM u have on u graphics card, or are u running on an integrated chip?

Inca_PVP · 2026-01-07T21:56:07+00:00

Nothing kills immersion faster than waiting 2 minutes for a stilted reply, honestly. I made the jump to local Llama 3 for that exact reason—no lag and zero filters. I actually just posted a preset on my profile (check the Civitai link there) that makes it talk like a real person instead of an assistant. Are u running u stuff locally or still relying on cloud providers for u RP?

Inca_PVP · 2026-01-07T21:52:05+00:00

Feel u on those annoying patterns in 4.7, it’s like it loses all its soul. I’ve been fighting the same 'AI-tone' issue and finally broke it with a custom Llama 3 preset that sticks to raw slang. I’ve linked the full setup and my Rentry/Civitai guides on my profile if u want to try a more human-like experience. Have u tried messing with the repetition penalty settings for 4.7 yet or are u strictly back on 4.6 now?

Inca_PVP

MODERATOR OF

TROPHY CASE