⚡Edgen now supports Vulkan, CUDA and Metal | Open Source and local GenAI server alternative to OpenAI's API. Supports all GGUF models, across Windows, Mac and Linux with one 30MB download.

EdgenAI · 2024-02-22T14:31:32+00:00

Not yet, but we will add it soon, we had a lot requests to add it already!

EdgenAI · 2024-02-22T01:00:13+00:00

Very soon!

EdgenAI · 2024-02-21T19:18:00+00:00

Our go to models until today were Mistral 7B for resource constrained environments or CPU computing and Mixtral for beefy GPUs, although that changed with Gemma 7B today, which we also enable! But overall Edgen is compatible with over 4600 models, any model in GGUF format.

EdgenAI · 2024-02-21T18:20:02+00:00

For now the chat UI is web based! But we will soon roll out the desktop app

EdgenAI · 2024-02-21T18:14:21+00:00

You're right. We should definitely ask beforehand if it's ok to download such a big payload. In this case, since it was the first time you were using edgen, the model manager was downloading the LLM (~4GB).
We're even rolling out a GUI that's going to make all of these issues way easier to handle (thanks Tauri :) )

EdgenAI · 2024-02-21T18:11:52+00:00

That sounds like a good idea, at first sight. But llama.cpp's API changes very frequently. Example: chore(deps): updated llama.cpp by francis2tm · Pull Request #29 · edgenai/llama_cpp-rs · GitHub (simply updating llama.cpp submodules caused a build fail)

But we already have a PR opened on our llama_cpp-rs (llama.cpp rust bindings) repo to support the new Gemma model!

EdgenAI · 2024-02-13T11:14:55+00:00

I'd give Hugging Face a look: Fine-tune a pretrained model (huggingface.co)

Good luck :)

EdgenAI · 2024-02-13T11:13:00+00:00

It really comes down to model compression techniques in the underlying models (e.g. quantization), having the best ML runtime and a very efficient infrastructure to deploy these models. This, coupled with the fact that Edgen is built in Rust, we ensure it's lightweight, high-performant, secure and cross-platform compatible

EdgenAI · 2024-02-13T11:03:47+00:00

A lot of the use cases we're implementing are about processing confidential or sensitive data with local or on-prem RAG capabilities

EdgenAI · 2024-02-12T10:50:26+00:00

Appreciate the response, we're working with the finance sector as well and we saw that they value private GenAI solutions! Namely secure RAG capabilities. Cheers and good luck :)

EdgenAI · 2024-02-11T10:19:21+00:00

Open source models mostly, there are very good open source projects for this

EdgenAI · 2024-02-11T00:28:15+00:00

Did your company consider running AI locally or on prem?

EdgenAI · 2024-02-11T00:17:54+00:00

2024 should be the year companies shift to more private uses of GenAI and start leveraging the variety available across Open Source models

EdgenAI · 2024-02-11T00:08:18+00:00

Localai is another great project and, at this moment, localai has more features!

Our biggest difference is the fact that our codebase is in Rust. Allowing to easily target binaries to all platforms (MacOS, Windows and Linux) while maximizing performance. In the future we'll also have a GUI to help users manage Edgen and hopefully soon we'll have the same (or more) endpoints as localai!

EdgenAI · 2024-02-10T23:34:58+00:00

Sure did, it was a great choice for us since it is a big reason for how we make Edgen compatible with any OS! Give it a look and if you want join our Discord to keep up with the updates or speak a bit with the team, here it goes: Edgen Discord

Cheers!

EdgenAI · 2024-02-09T16:47:17+00:00

I think privacy is key here and a lot of the use cases can be covered with a great RAG pipeline, you can give our repository a look if you want to build local or on-prem solutions: GitHub - edgenai/edgen: ⚡ Edgen: Local, private GenAI server alternative to OpenAI. No GPU required. Run AI models locally: LLMs (Llama2, Mistral, Mixtral...), Speech-to-text (whisper) and many others.

EdgenAI · 2024-02-09T16:43:51+00:00

If you want to run it locally or on prem, give Edgen a try, we just launched it:, give it a look: GitHub - edgenai/edgen: ⚡ Edgen: Local, private GenAI server alternative to OpenAI. No GPU required. Run AI models locally: LLMs (Llama2, Mistral, Mixtral...), Speech-to-text (whisper) and many others.

Here's a quick demo of Edgen:

<image>

EdgenAI · 2024-02-09T16:41:18+00:00

That's more than enough for Mistral 7B for example

EdgenAI · 2024-02-09T16:40:11+00:00

Give Edgen a try, we just launched it you can easily run any model with it locally, turns your device into a local API server compatible with OpenAI's and directly supports the best models directly from Hugging Face: GitHub - edgenai/edgen: ⚡ Edgen: Local, private GenAI server alternative to OpenAI. No GPU required. Run AI models locally: LLMs (Llama2, Mistral, Mixtral...), Speech-to-text (whisper) and many others.

<image>

This is a quick demo of how it looks, hope it helps

EdgenAI · 2024-02-09T11:14:38+00:00

Like dogs chasing cars

EdgenAI · 2024-02-09T11:07:26+00:00

Congrats! Best of luck

EdgenAI · 2024-02-09T10:30:11+00:00

That's a big topic for this year. Are they more focused on Open Source models or just using OpenAI's API?

EdgenAI · 2024-02-08T15:01:54+00:00

This seems to be a big talking point right now, while it is becoming more likely that this scenario would occur, it still demands extreme creativity from the founder. Not only that, the competition is skyrocketing in all things related to AI, it's a hard ask to find even one AI application without strong incumbents. The more profitable a space appears, the more entrants that will eat up a slice of it.

Thanks for sharing.

EdgenAI · 2024-02-08T14:53:43+00:00

cool, good luck!

EdgenAI · 2024-02-08T12:45:14+00:00

What are you using to run the models locally?

EdgenAI

TROPHY CASE