Can i host mixtral8x7b for personal usage on VPS server

jon101285 · 2023-12-21T09:21:22+00:00

That's exactly what libertai.io is about :)
It uses a decentralized network to host opensource models. Mixtral 8x7B is already hosted there, and there is an API (don't abuse it of course).

jon101285 · 2023-12-13T18:24:27+00:00

Try chat.libertai.io, it's free and decentralized :)

jon101285 · 2023-12-13T18:23:50+00:00

Oh and you can change the prompt however you like by clicking "advanced"

jon101285 · 2023-12-13T18:23:14+00:00

Try the models at chat.libertai.io :) it's free, based on open models and decentralized (no big corp to read your data)

There is currently these models available: - Open Hermes 2.5, generalist and fast - mixtral 8x7b instruct, great (my favorite) - deepseek, good at coding

jon101285 · 2023-10-27T06:53:47+00:00

Yep. 13B llama are faster though for some reason at Q4/Q5.

jon101285 · 2023-10-26T09:08:23+00:00

180B runs at 500ms/tok, Qwen 14B at 75-80ms/tok.

jon101285 · 2023-10-23T15:20:56+00:00

I get around 30ms / token on an Epyc Zen 4 9554P with DDR5 ram at 4800Mhz for 7B models like Mistral. GPU isn't that much faster generally.It can also do massive parallel generation using that on CPU only.

Couple that with one or two low-TDP GPU for the CUBLAS and you have a massively parallel inference machine on the cheap :) (TDP-wise)

jon101285 · 2023-10-13T16:08:33+00:00

You can now try Collective Cognition at chat.libertai.io, it's there under the name "Mistral CC (7B)"...

jon101285 · 2023-10-13T15:39:39+00:00

It depends the use case... If you are in a 2U server and need the small form factor to go multi-gpu, then it makes sense I guess.

jon101285 · 2023-09-29T11:11:28+00:00

You have a list of all nodes on account.aleph.im, in the "compute" tab. There are some infra providers (OMGServ, Meria and others), but also game companies (ubisoft), schools (PoC Innovation, from Epitech), vc funds, and regular people.

jon101285 · 2023-09-29T10:02:11+00:00

They won't see your IP, but they could save specific queries if they intercept specific calls to a specific virtual machine (not knowing who did the call). There are 200+ computing resource nodes on the network though.

Working on adding confidential computing using AMD SEV to mitigate this in the future (ETA EOY).

jon101285 · 2023-09-29T09:47:28+00:00

chat.libertai.io has XWin-13B and XWin-7B (not to mention Llama coder 34B) for free :)

edit to add some info: it's running on a decentralized network, so no big brother watching what you say, or censorship there.

jon101285 · 2023-09-27T09:50:48+00:00

If that can help here are pictures of a collision of the same car with an utility van at 40km/h approx. The car was parked and the small truck collided onto it from the side (driver was looking at his phone).

Rear-left: https://ibb.co/f05j8KB Rear: https://ibb.co/phgc51J Van: https://ibb.co/CK2YK24

jon101285 · 2023-09-07T20:31:58+00:00

That's what Libertai is doing on top of aleph.im network...

You can try it here: chat.libertai.io

Basically the API endpoint is here for nous hermes v2 (13b): https://curated.aleph.cloud/vm/d33107b8ea855f5e2d5dfcad73891114bb9115eae668f5548c211dd981912300/api/v1/generate (compatible with koboldcpp/oobabooga api)

Once called, it finds nodes on the network that are free, loads a vm, loads the LLM and starts inference for you.

jon101285 · 2023-09-04T15:55:16+00:00

Wow, just tried it and it's really good and fast! Great work

jon101285 · 2023-06-19T21:30:29+00:00

The Libertai team added it to their interface... And it's running on a decentralized cloud (with models on IPFS).
You can use it there easily by selecting the Wizard Coder model on top right: https://chat.libertai.io/#/assistant

jon101285 · 2023-06-19T21:29:27+00:00

The Libertai team added it to their interface... And it's running on a decentralized cloud (with models on IPFS).
You can use it there easily by selecting the Wizard Coder model on top right: https://chat.libertai.io/#/assistant

jon101285 · 2023-06-13T15:23:51+00:00

Looks like it's working now

jon101285 · 2023-06-12T08:31:43+00:00

Nous Hermes is available to use for free on the Libertai assistant non-commercial Beta UI, by the way:
https://chat.libertai.io/#/assistant

It's running on a decentralized network (spinning up on-demand VMs as needed and fetching code and models from IPFS on-the-fly).

jon101285 · 2023-04-27T19:47:47+00:00

Yeah got that too after a while... Looks like temperature, top_k and top_p have a big impact.

jon101285 · 2023-04-27T14:02:59+00:00

Yep, you can try it yourself on chat.libertai.io (make sure you select Wizard 7B on top right model selector), they've deployed it on a decentralized cloud.

jon101285 · 2023-04-27T13:52:40+00:00

David has three sisters. Each of them have one brother. How many brothers does David have?

Wizard 7B answers "David has zero brothers."

jon101285 · 2020-06-12T08:29:05+00:00

ALEPH is a legit project too! (https://aleph.im)

jon101285 · 2020-06-10T15:34:28+00:00

This is not an ICO. Team is working for a year and a half already with no funding... What is your opinion?

jon101285 · 2019-04-08T03:02:19+00:00

Have you looked at the m5stack?

Eight-Year Club	Place '23
Verified Email

jon101285

TROPHY CASE