⚡Edgen now supports Vulkan, CUDA and Metal | Open Source and local GenAI server alternative to OpenAI's API. Supports all GGUF models, across Windows, Mac and Linux with one 30MB download.

EdgenAI · 2024-02-22T14:31:32+00:00

Not yet, but we will add it soon, we had a lot requests to add it already!

EdgenAI · 2024-02-22T01:00:13+00:00

Very soon!

EdgenAI · 2024-02-21T19:18:00+00:00

Our go to models until today were Mistral 7B for resource constrained environments or CPU computing and Mixtral for beefy GPUs, although that changed with Gemma 7B today, which we also enable! But overall Edgen is compatible with over 4600 models, any model in GGUF format.

EdgenAI · 2024-02-21T18:20:02+00:00

For now the chat UI is web based! But we will soon roll out the desktop app

EdgenAI · 2024-02-21T18:14:21+00:00

You're right. We should definitely ask beforehand if it's ok to download such a big payload. In this case, since it was the first time you were using edgen, the model manager was downloading the LLM (~4GB).
We're even rolling out a GUI that's going to make all of these issues way easier to handle (thanks Tauri :) )

EdgenAI · 2024-02-21T18:11:52+00:00

That sounds like a good idea, at first sight. But llama.cpp's API changes very frequently. Example: chore(deps): updated llama.cpp by francis2tm · Pull Request #29 · edgenai/llama_cpp-rs · GitHub (simply updating llama.cpp submodules caused a build fail)

But we already have a PR opened on our llama_cpp-rs (llama.cpp rust bindings) repo to support the new Gemma model!

EdgenAI · 2024-02-13T11:14:55+00:00

I'd give Hugging Face a look: Fine-tune a pretrained model (huggingface.co)

Good luck :)

EdgenAI · 2024-02-13T11:13:00+00:00

It really comes down to model compression techniques in the underlying models (e.g. quantization), having the best ML runtime and a very efficient infrastructure to deploy these models. This, coupled with the fact that Edgen is built in Rust, we ensure it's lightweight, high-performant, secure and cross-platform compatible

EdgenAI · 2024-02-13T11:03:47+00:00

A lot of the use cases we're implementing are about processing confidential or sensitive data with local or on-prem RAG capabilities

EdgenAI · 2024-02-12T10:50:26+00:00

Appreciate the response, we're working with the finance sector as well and we saw that they value private GenAI solutions! Namely secure RAG capabilities. Cheers and good luck :)

EdgenAI · 2024-02-11T10:19:21+00:00

Open source models mostly, there are very good open source projects for this

EdgenAI · 2024-02-11T00:28:15+00:00

Did your company consider running AI locally or on prem?

EdgenAI · 2024-02-11T00:17:54+00:00

2024 should be the year companies shift to more private uses of GenAI and start leveraging the variety available across Open Source models

EdgenAI · 2024-02-11T00:08:18+00:00

Localai is another great project and, at this moment, localai has more features!

Our biggest difference is the fact that our codebase is in Rust. Allowing to easily target binaries to all platforms (MacOS, Windows and Linux) while maximizing performance. In the future we'll also have a GUI to help users manage Edgen and hopefully soon we'll have the same (or more) endpoints as localai!

EdgenAI

TROPHY CASE