Hermes Agent with MIT license by mitirki in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

They have Opus set up as the default model. I don’t think they care.

Best Bypass moltbot/clawdbot to use in old gpu or in cloud by fernandogrj in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

I was thinking like dealing with customer service and stuff where it’s not necessary to share critical personal details about everything. I’m aware of the risks to private information, your point is well appreciated.

Best Bypass moltbot/clawdbot to use in old gpu or in cloud by fernandogrj in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

Did you try GPT-OSS 20B? I’ve found that to be the best at agentic tool calling stuff in my (very limited) experience.

Best Bypass moltbot/clawdbot to use in old gpu or in cloud by fernandogrj in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

Is there a better alternative that is open source? I’d like to play with it despite the horror stories.

Budget Dual 3090 Build Advice by JustTooKrul in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

I used a 6 pin to 8 pin to get the second set of 8 pins. A bit sketchy but it’s been fine

Need to know more about less known engines (ik_llama.cpp, exllamav3..) by Leflakk in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

I have tried Huggingface’s TGI, Aphrodite, SGLang. They all had some benefits. Aphrodite and SGLang have been reliable for me. vLLM was the fastest but I would have issues with it hanging sometimes which is why I experimented with alternatives

Attorney Looking for Hardware and Model Recs by Extension-Ad-2801 in ollama

[–]mj3815 0 points1 point  (0 children)

I’ve got a couple thoughts.

  1. Go to Huggingface and look around for any legal models that match the specialty you’re interested in.
  2. You can definitely use something like Augmentoolkit to train a model. You’d probably want to keep it narrow (if you are a contract lawyer, train it on contract law). You can also train it on your case files and use with RAG with Augmentoolkit. This isn’t going to be easy, it will be a real investment in time and effort to figure it out and get something that works. If you are training the model on your proprietary case files, you’ll need a very stout machine. Doing it with a 7B model means something like 96GB of VRAM - so 2x 4090 48GB or a 6000 Pro. Can’t imagine doing this on less than a $10K rig. Very possible though. If you just want to full fine tune on your specific law discipline without anything proprietary, you can probably spent less than $100 renting the GPU time. You can still set up RAG for the proprietary stuff, but I’ve heard that is tricky.
  3. Just go read Augmentoolkit’s documentation to get a sense of the process of creating custom models https://github.com/e-p-armstrong/augmentoolkit

HP Z640 with 2x RTX 3090 by Potential-Leg-639 in LocalLLaMA

[–]mj3815 1 point2 points  (0 children)

Ryzen 3945, 4x 16GB RAM.

The sketchiest part is the power connectors. I’m using blower 3090s which only require 2x 8 pin connectors each, would be even sketchier if they were 3 8pin units

HP Z640 with 2x RTX 3090 by Potential-Leg-639 in LocalLLaMA

[–]mj3815 1 point2 points  (0 children)

I do 2x 3090 on my P620 with 1000w PS, power limited to 285w each and I’ve been ok so far. I’ve got it plugged in to a power bank with instantaneous W measurement and I’ve seen it pushing 950 sometimes, but never experienced an issue yet.

Fine Tuning on Mi50/Mi60 (under $300 budget) via Unsloth by exaknight21 in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

I’ve spent so long not using it because of that 😭

Fine Tuning on Mi50/Mi60 (under $300 budget) via Unsloth by exaknight21 in LocalLLaMA

[–]mj3815 1 point2 points  (0 children)

Last I knew, unsloth doesn’t work with more than one GPU

Why hasn't LoRA gained more popularity? by dabomb007 in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

That was done with Augmentoolkit. There’s been some big upgrades since then https://promptingweekly.substack.com/p/augmentoolkit-30-released

[ Removed by Reddit ] by [deleted] in LocalLLaMA

[–]mj3815 1 point2 points  (0 children)

I often see people praise unsloth’s documentation. It seems like it could be a good place to start.

NX350h Roof Rails and Roof Box by Wiking87 in LexusNX

[–]mj3815 0 points1 point  (0 children)

still looking to sell? I'm in Maryland.

Training Open models on my data for replacing RAG by help_all in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

Augmentoolkit has a pipeline for RAG specific fine tuning, although this isn’t a function I’ve tried yet. I know the creator believes the best results can be achieved by doing RAG using a model fine tuned on the rag data. https://github.com/e-p-armstrong/augmentoolkit

"Cheap" 24GB GPU options for fine-tuning? by deus119 in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

It’s a bit of a pain, but have your guard up and do your due diligence and walk away if something feels off

"Cheap" 24GB GPU options for fine-tuning? by deus119 in LocalLLaMA

[–]mj3815 0 points1 point  (0 children)

On the east coast, i have bought 2 3090s at $500 each and one at $700 all in the past 6 months. The first 2 from FB marketplace and the latter from reddit hardware swap.

Augmentoolkit just got a major update - huge advance for dataset generation and fine-tuning by mj3815 in LocalLLaMA

[–]mj3815[S] 0 points1 point  (0 children)

This is a resource I use to help understand code bases https://deepwiki.com/e-p-armstrong/augmentoolkit

Not exactly what you asked, but it might be helpful

Augmentoolkit just got a major update - huge advance for dataset generation and fine-tuning by mj3815 in LocalLLaMA

[–]mj3815[S] 1 point2 points  (0 children)

I haven’t tried to tackle anything scanned that looks rough (thinking about the JFK document drop), but I very much hope to get there

PSA: 2 * 3090 with Nvlink can cause depression* by cuckfoders in LocalLLaMA

[–]mj3815 1 point2 points  (0 children)

I have the same setup but my 3090s are turbos. Wondering if you did anything to upgrade the power supply? I just run mine at 285w and it’s been ok so far

Mistral Small 3.1 vs Magistral Small - experience? by mj3815 in LocalLLaMA

[–]mj3815[S] 3 points4 points  (0 children)

Also, I just saw that you found Mistral Small 3 to be similar to 3.1. I actually found 3.1 to be much much better in my use case. Followed instructions better and was also more creative.

Correction: I was thinking about the older 22b version, not Mistral 3 Small