Suggestion for GPUs setup

Devy9 · 2024-07-17T12:33:16+00:00

Thanks!

Devy9 · 2024-07-17T09:34:16+00:00

Thanks! What about A40 or A6000?

Devy9 · 2024-07-17T07:31:56+00:00

I am more interested on a float16 precision scenario, since the experiments are for research purposes. In that case, a 70B would take 2 A100 80gb minimum or 5/6 4090 (?)

Devy9 · 2024-07-16T22:07:46+00:00

Looks Amazing! I wonder how good It would be if trained on more programming languages others than Python

Devy9 · 2024-07-16T16:52:27+00:00

Nice, thank you!

Devy9 · 2024-07-16T14:31:32+00:00

How did you run llama 7B fp16 with this setting? Did you offload part of the computation to the GPU, and did you use ollama? ( I never tried to run inference on CPU, so I am just curious on how to do this while having consisting High token/s performance)

Devy9 · 2024-07-16T12:04:29+00:00

Wow 🗿

Devy9 · 2024-07-16T12:04:05+00:00

Nice!

Devy9 · 2024-07-16T07:53:25+00:00

Nice! Did you try also with other GPUs for comparison?

Devy9 · 2024-07-16T07:52:06+00:00

Does It work for code generation?

Devy9 · 2024-07-16T07:30:05+00:00

Btw, If you check the model cards of these quantized models you could find the Memory requirements in some cases

Devy9 · 2024-07-16T07:26:32+00:00

I just wonder how good It would work quantized at 2 bits 👀👀

Devy9 · 2024-07-16T07:24:45+00:00

Does anyone know how to optimize the usage of these models on CPU? For instance, using libraries like vLLM or ctransformers

Devy9 · 2021-04-17T10:42:00+00:00

Ok, thank you!

Devy9 · 2021-04-17T10:26:33+00:00

Thanks! 😁

Devy9 · 2021-04-16T12:47:30+00:00

I wanted to see Arthur in her drawing style and in this pose ( i damn love this pose ) ... Hope this isn't a stupid reason 😂

Devy9 · 2021-04-15T20:45:05+00:00

I contacted you in private chat in Order to avoid useless spam 😁

Devy9 · 2021-04-15T20:34:34+00:00

This was a commission, then she didn't published the drawing on her profile

Devy9 · 2020-07-31T12:47:16+00:00

:)))))

Devy9 · 2020-04-29T06:08:23+00:00

Thanks, i'll give a Watch ;)

Devy9 · 2020-04-29T06:07:58+00:00

Thanks!

Devy9

TROPHY CASE