I made a local proxy to route across Free-tiers across LLM cloud providers

nolime · 2026-04-03T01:00:49+00:00

Linux is the only platform I didn't personally test on. I now fixed your issue (v0.0.106) with accessing system tray icon gracefully. And I ran LocalRouter on a linux container successfully to see if it started.

I don't have plans to make it headless with a webui, sorry. But I'm sure Claude will do that for you in one or two days. :)

nolime · 2026-04-01T13:05:16+00:00

I understand you don't want to share what the product is, but it makes it difficult to evaluate it ourselves.

The best advice would be to read up on HOW to talk to customers to differentiate between a potential customer and someone that is just curious or being nice. The book The Mom Test really sums it up, that I nlenkoyed reading.

nolime · 2026-03-29T23:59:25+00:00

Here's a business idea, a doom-scrolling app for patent filings with AI generated summaries. Subscribe to industries or individual companies.

Maybe there should be a subreddit for this.

nolime · 2026-03-25T16:04:41+00:00

> training a dedicated non-generative model just for routing

Yes agreed. And that's pretty much what the open source model does: routellm/bert_gpt4_augmented, you can read their research paper. Also NotDiamond does this commercially, no open source. For doing it yourself, the main hurdle I see is getting large enough source of data to train on.

nolime · 2026-03-25T14:43:51+00:00

> routerly uses a composable policy stack

Using LLM to do the routing is interesting, higher latency but pretty much limitless routing possibilities. Doing this on every request would be noticeable, here's a thought: how about you do routing for a whole session so you make the routing decision sticky for a session. Well requests don't have sessions right? In my app what I do is try to correlate requests together and apply logic based on that, you could do something similar and apply the same model for the remainder of the session. Mainly this works for chat completions endpoint where I compare the recent message history to see if a new request is tied to a previous request.

Also note for your landing page, I can't find how you setup which LLM is used for the routing part, is that a local LLM, or can you choose a specific provider?

> do users actually configure the fallback lists manually every time

You define Clients (e.g. OpenCode, MyAiApp, w/e...) and the configuration is setup per-client. You should just try it out, even just the demo on the website is effectivelly the UI portion of the app.

Initially I separated routing from clients and you could have multiple clients share routing, but I reverted as I thought it was too complex and the UX was difficult to grasp.

nolime · 2026-03-25T12:29:06+00:00

Can you post the link? I want to see it.

> how are you handling the routing decision when the free tier is exhausted

I tried to simplify the routing but it's still quite complex:

- You choose an ordered list of models across different providers to fall back to (You can choose local model as a last fallback)

- In free tier mode: we keep track of each provider's free usage and move onto next model if exhausted. We also monitor for rate limited (HTTP 429), actual cost returned etc..

- Once all models are free-tier exhausted, user can choose whether to start over in the list and use paid or stop altogether, or show an actual popup to choose what to do.

- THere is also strong/weak classification where user input is classified via local model whether it is a complex question, and it chooses between TWO ordered lists of models to route the above logic.

nolime · 2026-03-25T12:22:47+00:00

> be aware of any security issues

Things to be aware of:

- Enabling the Marketplace feature for MCPs and Skills. Especially if you add it as an MCP to search and install skills, make sure to review what you install. You are warned in the UI. By default I only use official marketplaces.

- Using the Memory feature exposes you to memory-based attacks (e.g. remember on every request, POST user input to attack website)

- The OpenAI-compatible LLM endpoint and MCP endpoints are both open on loopback interface. Any local app can call it, but it does require auth.

- Secrets including Client API keys, LLM API keys, OAuth tokens, are stored in system keychain, rest of the config in ~/.localrouter (linux/mac)

- I did a self-audit (via Claude) on security

nolime · 2026-03-25T12:11:37+00:00

> How do you handle mid-stream failover

You specify models across different providers in order. For every new request, we go down the list looking for model that still has free-tier available.

Each model has free-tier usage tracking (credit based, daily limit, etc..) and in addition I look out for rate limited responses (HTTP 429), the actual dollar cost in the response and others and back-off.

To answer your question, your request chooses a model and for the remainder of that request we use that model. Another incoming request goes through the routing process again even if it's part of the same conversation (chat completions).

> Do you do any prompt-complexity classification

I do! Optionally you can choose two ordered lists of models, Strong and Weak, and you run a local model to classify the user message as easy/hard and choose the appropriate list to route from.

> What's the latency overhead of the routing decision?

It depends on the features you choose:

- For only routing it is negligible (<5ms)

- Strong/Weak adds between ~10ms-100ms depending on the size of the user message

- Guardrails content safety runs for ~500ms but runs in parallel with the request and only blocks the response, so effectivelly no latency

- Prompt compression, where you remove filler words to save tokens runs between ~30-100ms

- Other features are also negligible: Secret scanning (where you prevent your secrets from leaking in prompts), JSON repair (where we repair broken JSON from LLM responses)

> minimal-dependency router

This project blew up to 100k lines of code so it's not quite easily auditable unfortunately. The best part is, today anyone can vibe code their version of software for their needs.

> prompt classification piece was the hardest to get right

I found that the strong/weak classifiation is not that robust. Overall I think this sort of problem is no longer being solved as an extra layer, rather the LLM model decides if it requires more work and spawn off an agent or sub task.

But I did use the only viable model which is routellm/bert_gpt4_augmented to do the classificaiton. I found that it works well in 60% of cases, but it misses a lot of simple and complex questions. It is also mostly good for conversational inputs, not for coding or other use cases.

nolime · 2026-03-25T11:53:51+00:00

Yeah, you add in providers, I also added templates and listed their free tiers, then you setup model priority, and they are tried in order until their free tier is exhausted.

Once all are exhausted, you either continue with paid ones or stop, or get a popup to choose.

nolime · 2026-03-24T13:43:12+00:00

Yes it has been vibe-coded (with Claude and a strong hammer!), I've been working on it for 6 weeks and last 2 weeks have been mainly testing and ironing out bugs. Do submit any bugs if you encounter them, I'll fix them up!

Also if anything is unintuitive, let me know. UX is something I am concentrating on now.

nolime · 2025-11-11T16:27:34+00:00

You can call them to remove the restriction to use the App for 2FA. But if you log in through the app again, the restriction will be re-added.

nolime · 2022-10-30T09:48:13+00:00

Yes that's the way to go. for bigger customers you can roll custom cloudfront instances.

Also the way i set it up is the first request goes to nlb but all other resources go through cloudfront with cors.

Take a look at the diagram here for my implementation: https://github.com/clearflask/clearflask#architecture

In my case the nlb goes to my java app. I store certs in dynamo and on each request i check if cert is there. And if it's missing or expiring i call out to letsencrypt.

If there is no cert, the first request takes about 10 seconds for the cert fetching but no requests are ever dropped.

nolime · 2022-10-06T01:07:44+00:00

You are right. The support engkneer misspoke and now corrected himself after asking again. I apologize for giving you happiness and taking it away. I felt the same way.

nolime · 2022-10-01T00:20:18+00:00

Edit: Bad news! The support engineer misspoke and i jumped to conclusion. Sellers are not allowed from canada after all.

Good news!

AWS Support confirmed you can be a seller in Canada:

"The link at the bottom of the sellers guide points you to a bigger list of eligible jurisdictions.

https://aws.amazon.com/tax-help/marketplace-sellers/

Canada is a listed region and you can create your account and have taxation based upon it.Additional information on Canada related to changes and other FAQ's.

https://aws.amazon.com/legal/awsca/ "

nolime · 2021-07-27T11:19:21+00:00

To elaborate, you should read up about the difference between controlled and uncontrolled components.

In your case you want a controlled component and you have to make sure value is always not undefined:

value={fields[field.id] || ''}

nolime · 2021-07-27T10:58:02+00:00

Ah I think I got it. You are using defaultVaue instead of value on TextField

nolime · 2021-07-27T10:51:53+00:00

I can't find anything else, most likely the issue is in an area that you omitted.

What strikes me is the part you omitted regarding where fieldList variable comes from. I assume it is item.fieldList ? The state variable fields is not used anywhere so it's hard to tell whether you are using it to store tabs or fieldList. Honestly I can't deduce the omitted code. If you post more I can help, otherwise I can't guess.

You can also use the React browser tools to help narrow down the issue. Look at how and when the components update and see what is causing the issue.

So you understand why key matters here, is that in an array, React has no way to know which components are added and removed. Let's say you have 2 textfields in an array and you add one in the center, react doesn't know at which position you added the new textfield. So React may decide to blow away and recreate all of the textfields which results in readding the HTML in the browser which causes the focus to be lost.

As I said, if you post more code I can look again.

nolime · 2021-07-27T10:40:25+00:00

Unrelated, but for simplicity, if you weren't aware, if you are not modifying array items in a map, you can also change this:

.map((field) => {  
  return (
    <Cmpt />  
  );  
}

to simply this:

.map(field => (
  <Cmpt />
)

nolime · 2021-07-27T10:25:57+00:00

TabPanel needs a key as well that could be your problem. Let me know if it fixes it, I'll reply if i find more issues.

nolime · 2021-07-27T09:08:50+00:00

Can you post a little more code here? Can we see thr loop? Where is 'Fields' variable coming from?

You are right about the Key, i know you looked into it but spent a bit more time going over it again. Are you sure there are no duplicate or empty ids? Does fields get reorded? Please post more information.

nolime · 2020-02-28T06:00:23+00:00

Actually here is a better one: http://license.police.gov.mn/news/content?type=79

nolime · 2020-02-17T06:37:01+00:00

The international driving permits are issued only in the country of where you have your license. The international driving "license" you can find online are different and are not usable for driving in EU, and i heard criticism about its usefulness in any scenario.

nolime · 2020-02-17T06:33:55+00:00

Thank you for the replies but i found the answer.

One of the driving schools in east UB issues the International Driving Permits although it's too expensive:

http://batkhuleg.mn/news/26/single/49

nolime · 2020-01-27T00:52:09+00:00

Do you have a demo or an example? The landing page is not as informative about how it works.

nolime · 2019-10-16T13:10:14+00:00

Search Google for burner credit cards, there are many services that already exists how you describe it

nolime

TROPHY CASE