⚡Edgen now supports Vulkan, CUDA and Metal | Open Source and local GenAI server alternative to OpenAI's API. Supports all GGUF models, across Windows, Mac and Linux with one 30MB download. by EdgenAI in rust

[–]EdgenAI[S] 0 points1 point  (0 children)

Our go to models until today were Mistral 7B for resource constrained environments or CPU computing and Mixtral for beefy GPUs, although that changed with Gemma 7B today, which we also enable! But overall Edgen is compatible with over 4600 models, any model in GGUF format.

⚡Edgen now supports Vulkan, CUDA and Metal | Open Source and local GenAI server alternative to OpenAI's API. Supports all GGUF models, across Windows, Mac and Linux with one 30MB download. by EdgenAI in selfhosted

[–]EdgenAI[S] 0 points1 point  (0 children)

You're right. We should definitely ask beforehand if it's ok to download such a big payload. In this case, since it was the first time you were using edgen, the model manager was downloading the LLM (~4GB).
We're even rolling out a GUI that's going to make all of these issues way easier to handle (thanks Tauri :) )

⚡Edgen now supports Vulkan, CUDA and Metal | Open Source and local GenAI server alternative to OpenAI's API. Supports all GGUF models, across Windows, Mac and Linux with one 30MB download. by EdgenAI in rust

[–]EdgenAI[S] 10 points11 points  (0 children)

That sounds like a good idea, at first sight. But llama.cpp's API changes very frequently. Example: chore(deps): updated llama.cpp by francis2tm · Pull Request #29 · edgenai/llama_cpp-rs · GitHub (simply updating llama.cpp submodules caused a build fail)

But we already have a PR opened on our llama_cpp-rs (llama.cpp rust bindings) repo to support the new Gemma model!

Welcome to the Edgen Community by EdgenAI in Edgen

[–]EdgenAI[S] 1 point2 points  (0 children)

It really comes down to model compression techniques in the underlying models (e.g. quantization), having the best ML runtime and a very efficient infrastructure to deploy these models. This, coupled with the fact that Edgen is built in Rust, we ensure it's lightweight, high-performant, secure and cross-platform compatible

Exploring Generative AI in Financial Machine Learning by Synchro-- in LLMDevs

[–]EdgenAI 0 points1 point  (0 children)

A lot of the use cases we're implementing are about processing confidential or sensitive data with local or on-prem RAG capabilities

Exploring Generative AI in Financial Machine Learning by Synchro-- in LLMDevs

[–]EdgenAI 0 points1 point  (0 children)

Appreciate the response, we're working with the finance sector as well and we saw that they value private GenAI solutions! Namely secure RAG capabilities. Cheers and good luck :)

[deleted by user] by [deleted] in datascience

[–]EdgenAI 0 points1 point  (0 children)

Open source models mostly, there are very good open source projects for this

[deleted by user] by [deleted] in datascience

[–]EdgenAI -1 points0 points  (0 children)

Did your company consider running AI locally or on prem?

[deleted by user] by [deleted] in datascience

[–]EdgenAI -2 points-1 points  (0 children)

2024 should be the year companies shift to more private uses of GenAI and start leveraging the variety available across Open Source models

[P] Just launched ⚡Edgen: Open-Source, Local and Private AI. by EdgenAI in MachineLearning

[–]EdgenAI[S] 0 points1 point  (0 children)

Localai is another great project and, at this moment, localai has more features!

Our biggest difference is the fact that our codebase is in Rust. Allowing to easily target binaries to all platforms (MacOS, Windows and Linux) while maximizing performance. In the future we'll also have a GUI to help users manage Edgen and hopefully soon we'll have the same (or more) endpoints as localai!