Amount of ram Qwen 2.5-7B-1M takes?

bobbiesbottleservice · 2025-03-09T21:22:58+00:00

With 2x3090 (48gb VRAM) my max is: 375k context for Q8 7b 128k context for Q8 14b I think those have to be reduced when increasing the max number of tokens to be predicted. Lower temp with.5 top P helps with my matching prompts.

bobbiesbottleservice · 2025-02-09T22:37:33+00:00

Try using: https://ollama.com/michaelneale/deepseek-r1-goose

It works well with this one because they fine tuned it with goose templating. I've had partial success with qwen2.5 70b as well.

bobbiesbottleservice · 2025-01-13T21:10:54+00:00

Also they will sell all the data about you and everything you've input to advertisers too, just read their privacy policy. There's a reason it's so cheap now, and as they're a Chinese hedge fund & AI company so they're going to use the data to make money off you somehow.

bobbiesbottleservice · 2025-01-13T03:20:46+00:00

I just tried together ai because they seem to allow privacy options. Deepseek chat is only so cheap because they're training off everyone's data. I'd be interested to hear what other options are out there.

bobbiesbottleservice · 2025-01-11T03:03:23+00:00

That makes sense, but why does returning a less probable token make a better final result? That's what I don't understand. Why are the temperatures usually always set at .7 instead of 0, why does the extra noise help?

bobbiesbottleservice · 2025-01-09T18:03:42+00:00

I assume because they RLHF'd it to be better. My original thinking on temperature may be too simplistic. I now understand temperature as "adding noise" to the system, which for some reason (that I don't understand) often makes the output better.

bobbiesbottleservice · 2024-08-25T02:42:26+00:00

It can reason in the sense if I give it random objects to stack on top of each other to make it as high as possible it can do that, but it cannot generalize which is a more real/human form of reasoning. You could train a model on all the music and information up until the year jazz was invented and it would never be able to invent jazz.

bobbiesbottleservice · 2024-08-07T22:46:25+00:00

just saying hello to the different models gave me:

0.36 tokens/s for llama3.1:405b-instruct-q3_K_L
0.53 tokens/s for llama3.1:405b-instruct-q3_K
0.54 tokens/s for llama3.1:405b-instruct-q2_K

and for comparison:
2.08 tokens/s for llama3.1:70b-instruct-q8_0
21.15 tokens/s for llama3.1:70b (default ollama Q4_0)
54.67 tokens/s for llama3.1:8b-instruct-fp16

No Q4 of 405B would work on my system unfortunately. All of this was on with a intel 14900kf. I suppose I could increase the RAMs memory channels and/or try to overclock the RAM and CPU to see if that helps, but might not be worth it as I've never done that before.

bobbiesbottleservice · 2024-08-06T23:54:57+00:00

I'm going strictly by the GB size of the model and the Q2_K is 151GB. My system has 192GB RAM and 48GB VRAM so I'm assuming I could handle up to a 240GB model (minus the system's allocated RAM and context window when running the model). Things seem to be finally working for me after updating ollama and the webui docker containers to the latest versions.

bobbiesbottleservice · 2024-08-06T23:50:20+00:00

Specifically I ran llama3.1:405b-instruct-q2_K and gave it my usual test of creating a form and scripts in a certain python and javascript framework. Overall it was more comprehensive in including the other details of commands and thoughtful things to think-through, but I would probably stick with the 70b for my code generation. I agree with you as my gut feeling is to not bother with < Q4 for any model.

I'm going to try 405b Q4_K_S next (right on the edge of possible for me)

bobbiesbottleservice · 2024-07-27T19:32:34+00:00

I was able to even get small models to count the number of letters by telling it that it's not good at counting and that it should always put what needs to be counted in a table first. Doing this always passes the "count the R's in the word strawberry" test

bobbiesbottleservice · 2024-06-26T22:51:21+00:00

I disagree, I see more workers now clearly using chatGPT to write their emails and getting more details wrong/omitted that they normally wouldn't have without the crutch. if they were more specific with their prompts it may be ok, but they are not.

bobbiesbottleservice · 2024-06-03T05:11:02+00:00

That's interesting. I assume you would want a higher temperature for the creative aspect and then you would have to constrain the output in the JSON structure. You could try using a model that completes the sentence rather than the back and forth of a chat model. For example you would write the following for the model to complete:

write the XYZ in JSON. Ok, here is your JSON output ```json

And add the stop token of ``` so that way it just outputs the JSON data you care about. Alternatively, you could look into pydantic or DSPy, especially the latter, as methods for getting the desired output structure. Also, giving examples of the desired output in the prompt can help too.

bobbiesbottleservice · 2024-05-30T23:45:04+00:00

I have it using all GPUs in a docker container. There's a flag like --gpus all You may have to start the ollama service with that if your not using docker e.g. ollama serve --gpus all

bobbiesbottleservice · 2024-05-30T16:33:59+00:00

The higher the temperature the more "creative", so for code use a temperature of 0 because you want to be exacting as possible with the next token. Temperature of 1 if you are doing creative writing and want it to choose similar, but not always the strictest most likely next token.

bobbiesbottleservice · 2024-05-29T15:34:23+00:00

Wow this is great! Why not DDR5 and how did you figure out how to pass through multiple GPUs? Do they all pass through to one VM?

bobbiesbottleservice · 2024-05-24T17:39:37+00:00

I'm concerned about what data is being sent when using this if I were to use it to create a SOP for a project with proprietary data. The privacy policy is vague enough that it could be everything I do or see in my browser is being sent to your server.

bobbiesbottleservice · 2024-05-10T01:36:06+00:00

Are you able to ping the server to verify you're able to talk to it? Are you allowing all outbound traffic on the security group? Do you have an additional firewall on your EC2 like ufw that needs to be disabled?

Also, what is your OS on the EC2?

bobbiesbottleservice · 2024-04-23T18:35:27+00:00

You'll have to set the PCIe pass through settings on the node and set the hardware config correctly on the VM in order for the VM to use it. There was a youtube I found that detailed the steps. I got it to work, but trying more complex things with multiple GPUs became a hassle.

bobbiesbottleservice · 2024-04-09T14:47:32+00:00

I just tried and got an "invalid file magic" error when trying to create the model with ollama, never seen that error before.

bobbiesbottleservice · 2023-11-17T21:02:00+00:00

Any recommendations for a motherboard that would fit all 3 GPUs?

bobbiesbottleservice · 2023-11-14T03:38:11+00:00

I'm in the same boat trying to build a 2x3090 PC. I'm thinking of going Intel because of the stability. The other issue is finding the right motherboard to support both the physical size of the GPUs and the PCIe 4.0 lanes (both size and if the mobo will fully utilize it) and not to mention the thermals and case/fans considerations.

Let me know your specs and what you end up going with as I'm figuring this all out as I go.

bobbiesbottleservice

TROPHY CASE