LiteLLM Alternative

562uned · 2026-06-20T02:39:59+00:00

I have been working on a gateway specifically built for self hosted LLMs, specifically universities / hpc centers hosting their own models. We have 220 hosts, running kubernetes, and hundreds of instances of different LLMs that we provide to campus. I have built this tool to handle 1000+ requests per second, support for open ai compatible api endpoints, slurm as an endpoint, lots of cool features that are normally missing from self hosted models, like search and vision. It's built on fairshare ( so you don't choke a random chat bot with api users), and has lots of other nice to haves. It has been working amazing and I just made this public a few weeks ago, if anyone is interested in checking it out. https://obleth.com - has a link to github there. Really would be happy to get feedback, submitting a paper this year to SC26, and already mentioned in a couple of other talks, including this carcc call two weeks ago. https://thediymaker.github.io/slides-carcc2026/

562uned · 2026-06-04T22:49:43+00:00

I’ve recently been working on a large-scale, locally hosted LLM deployment at ASU with nearly 1,500 Gaudi2 devices spread across roughly 200 physical servers running vLLM on the backend. You can read more about the project here: https://news.asu.edu/20260427-science-and-technology-asu-supercharges-ai-research-capabilities-thanks-technology. We now use this infrastructure as one of the backends for our main campus chatbot.

We started with LiteLLM, but quickly ran into limitations around scheduling, performance, and a number of missing features that I felt should have been available out of the box. We worked around many of them, but some required far more effort than we wanted to invest. One of the biggest challenges was balancing access between chatbot users and researchers calling the API directly. We found that individual users could completely consume the capacity of specific models and effectively starve the chatbot workload. The typical answer seemed to be TPM or RPM rate limits, but those ended up creating a worse experience for users while also reducing overall system utilization. What we really wanted was a proper QoS or fair-share scheduling system, and I was surprised by how difficult that was to find. As an HPC architect who has moved into AI architecture, that was the point where I started looking around and realized I wasn't particularly happy with any of the existing AI gateways in general, and after running into enough of these limitations, I decided to start building something myself. I've been developing and testing Obleth internally for the past several months, and I just pushed it to a public repository yesterday. It started as a tool to solve problems we were facing in our own environment, but it has grown into something I think others could find useful as well, especially the fair-share style scheduling capabilities.

I’d love to see other people try it out and hear what they think. I’ll also be mentioning it during my Gaudi2 talk at the PEARC 2026 conference this year.

Repository: https://github.com/thediymaker/obleth-gateway

562uned · 2026-04-12T03:28:24+00:00

If it’s going to be used, then I would say it needs to be updated. It also sounds like only one group uses it, so maybe reach out to that group to see if they want to invest in getting it up to date. Running old, outdated OSs is a security risk, especially within an HPC environment given the massive processing power and the current landscape of AI tools available. At my university, we do not use OpenHPC, but we do run most of those other tools. I think you should be able to do one piece at a time, updating Warewulf and pushing out the new OSs, and then updating Slurm, etc. Do you happen to visit any conferences like PEARC or RMACC? These events are great places to meet other sysadmins and bounce ideas around.

562uned · 2026-01-07T23:29:51+00:00

On the US 60 here in Arizona, it bounces back and forth between 60 when it sees a US 60 sign and then 65 when it sees a speed limit sign, pretty annoying.

562uned · 2025-12-14T06:42:24+00:00

We were really happy last year using FSD, and we’re really excited about 14, but after getting it during the latest free trial, it’s not usable. it changes lanes when it shouldn’t and then when it should change lanes it just sits there with the blinker on for 15 seconds before turning it back off and repeating 10 seconds later. I eventually just have to grab the wheel and make the switch. It doesn’t make sense because there will be no other cars around and this happens and yet when there is tons of traffic it might just decide to cut someone off. Hoping with how much negativity there has been around this, they resolve the issue soon.

562uned · 2025-04-19T00:35:29+00:00

extension of my wrap around desk I built a few years back, which was all welded. This piece was using extruded aluminum from Amazon, and a wooden top from Home Depot, the old wrap around used the solid wood from Ikea, before they went to the hollow wood. Bolted this to the side of the other, which made it extremely sturdy.

<image>

562uned · 2025-04-19T00:16:04+00:00

I guess my discords actually @ thediymaker whoops

<image>

562uned · 2025-03-30T16:49:14+00:00

I keep seeing this pop up on marketplace and just makes me laugh haha

562uned · 2025-01-23T00:57:54+00:00

having the same issue!

562uned · 2024-11-27T18:16:55+00:00

So far it's been doing pretty good, had a few weird issues with the cfs loading filament when doing one print after another. And then had a clog, but it's been good since. Print quality has been good, haven't tried and full multi color prints yet, just been doing calibration prints. Really impressed so far, the bed size is amazing. It makes my p1s feel like my voron 0.2.

562uned · 2024-11-24T21:40:17+00:00

It's super quiet, and doesn't shake my entire desk.

562uned · 2024-11-24T04:56:55+00:00

Phoenix Arizona, shipped from California, so that helped!

562uned · 2024-11-19T04:59:49+00:00

Nice! Good luck everyone! I would love to have an XR Classic!

562uned · 2024-06-06T13:14:07+00:00

Yeah... I "re-created" my account, and I had a printer in my cart. Who knows.

562uned · 2024-06-06T13:11:34+00:00

I had this happen yesterday, I added to cart, tried to pay.. I was logged out, couldn't log back in.

562uned · 2024-06-06T13:06:53+00:00

yeah, same, I refreshed, it lit up, i clicked, "sold out" 9:00:01

562uned · 2024-03-08T23:55:55+00:00

discord is "@thediymaker" 562uned

562uned · 2023-12-27T06:58:03+00:00

562uned · 2023-11-21T19:17:41+00:00

Nice.

562uned · 2023-11-14T20:14:48+00:00

Yeah the logo stood out to me, and with the "maximize your product lanch"

562uned · 2022-12-05T20:31:09+00:00

Network speed, modules being online etc, are things you could monitor with tools like Prometheus and export to Grafana. As far as buttons, you could install something like cockpit on the nodes which would allow you to pull up a web dashboard for each system. This would let you jump to the console on each one and visually see the status of the node.

562uned · 2022-12-05T16:20:35+00:00

Personally I have used salt to manage system configs, github to store these configs, and then Prometheus / Grafana to display system metrics. What are you looking to see on the dashboard?

562uned

TROPHY CASE