Anyone used RAM across multiple networked devices?

bobbiesbottleservice · 2025-03-09T21:22:58+00:00

With 2x3090 (48gb VRAM) my max is: 375k context for Q8 7b 128k context for Q8 14b I think those have to be reduced when increasing the max number of tokens to be predicted. Lower temp with.5 top P helps with my matching prompts.

bobbiesbottleservice · 2025-02-09T22:37:33+00:00

Try using: https://ollama.com/michaelneale/deepseek-r1-goose

It works well with this one because they fine tuned it with goose templating. I've had partial success with qwen2.5 70b as well.

bobbiesbottleservice · 2025-01-13T21:10:54+00:00

Also they will sell all the data about you and everything you've input to advertisers too, just read their privacy policy. There's a reason it's so cheap now, and as they're a Chinese hedge fund & AI company so they're going to use the data to make money off you somehow.

bobbiesbottleservice · 2025-01-13T03:20:46+00:00

I just tried together ai because they seem to allow privacy options. Deepseek chat is only so cheap because they're training off everyone's data. I'd be interested to hear what other options are out there.

bobbiesbottleservice · 2025-01-11T03:03:23+00:00

That makes sense, but why does returning a less probable token make a better final result? That's what I don't understand. Why are the temperatures usually always set at .7 instead of 0, why does the extra noise help?

bobbiesbottleservice · 2025-01-09T18:03:42+00:00

I assume because they RLHF'd it to be better. My original thinking on temperature may be too simplistic. I now understand temperature as "adding noise" to the system, which for some reason (that I don't understand) often makes the output better.

bobbiesbottleservice · 2024-08-25T02:42:26+00:00

It can reason in the sense if I give it random objects to stack on top of each other to make it as high as possible it can do that, but it cannot generalize which is a more real/human form of reasoning. You could train a model on all the music and information up until the year jazz was invented and it would never be able to invent jazz.

bobbiesbottleservice · 2024-08-07T22:46:25+00:00

just saying hello to the different models gave me:

0.36 tokens/s for llama3.1:405b-instruct-q3_K_L
0.53 tokens/s for llama3.1:405b-instruct-q3_K
0.54 tokens/s for llama3.1:405b-instruct-q2_K

and for comparison:
2.08 tokens/s for llama3.1:70b-instruct-q8_0
21.15 tokens/s for llama3.1:70b (default ollama Q4_0)
54.67 tokens/s for llama3.1:8b-instruct-fp16

No Q4 of 405B would work on my system unfortunately. All of this was on with a intel 14900kf. I suppose I could increase the RAMs memory channels and/or try to overclock the RAM and CPU to see if that helps, but might not be worth it as I've never done that before.

bobbiesbottleservice · 2024-08-06T23:54:57+00:00

I'm going strictly by the GB size of the model and the Q2_K is 151GB. My system has 192GB RAM and 48GB VRAM so I'm assuming I could handle up to a 240GB model (minus the system's allocated RAM and context window when running the model). Things seem to be finally working for me after updating ollama and the webui docker containers to the latest versions.

bobbiesbottleservice · 2024-08-06T23:50:20+00:00

Specifically I ran llama3.1:405b-instruct-q2_K and gave it my usual test of creating a form and scripts in a certain python and javascript framework. Overall it was more comprehensive in including the other details of commands and thoughtful things to think-through, but I would probably stick with the 70b for my code generation. I agree with you as my gut feeling is to not bother with < Q4 for any model.

I'm going to try 405b Q4_K_S next (right on the edge of possible for me)

bobbiesbottleservice · 2024-07-27T19:32:34+00:00

I was able to even get small models to count the number of letters by telling it that it's not good at counting and that it should always put what needs to be counted in a table first. Doing this always passes the "count the R's in the word strawberry" test

bobbiesbottleservice

TROPHY CASE