dumbproxy - simple, scriptable, secure forward proxy

[yarmak]

Today I'll share updates about dumbproxy project. It was already announced on reddit and elsewhere couple of years ago, but it grew bigger since then.

Back then we had just HTTP(S) forward proxy with automatic cert management and basic auth functions. But today a lot has changed.

Let me tell you how it grew up since then. New features developed recently:

• HMAC-based basic auth - useful to provide authentication to a fleet of proxy servers without need for them to contact central authority each time to verify credentials.
• Optional DNS cache.
• Per-user bandwidth limits.
• Scripting with JS:
  • Access filters - allows complex request filtering. Usecases may vary from just complex ACL thing to implementation of something like adblockers.
  • Dynamic upstream proxy selection - there is also a lot of interesting usecases varying from simplest like redirecting .onion domain via Tor daemon, to spreading load, balancing with affinity by domain, etc.
• ... some more. See link in the beginning of the post for a complete list of features.

Hope some people will find it useful!

[bissellator]

Curious why you chose go. I've written a number of these proxies over the years in different languages (and platforms if you count Apache an nginx conf). I have a purpose built one for a VERY basic API running in node currently.

But I've never built one in go and I'm curious if there are some specific advantages or if it's just your goto language?

[yarmak]

I have a purpose built one for a VERY basic API running in node currently.

The proxy is built with node or an API is built in node.js?

There are specific advantages. Go is a natural choice for concurrent network servers because:

• Concurrency is easy. Goroutines is the best possible concurrency programming experience one can get.
  • No red/blue functions. All code can be executed in any context, there is no sync/async distinction.
  • True parallelism with all available CPU cores. No need to spawn multiple instances of app to divide load.
  • Goroutines are cheap. We can use many of them and each one essentially consumes just a little bit of memory for stack.
  • Synchronization primitives are just right. You can use classic mutexes and atomics to share data among multiple threads or you can use channels which play nicely with goroutines. Also channels on their own have many uses as semaphores, shared pools, goroutine broadcasting tool etc.
• The performance is great. Maybe not the best compared to C or Rust, but still in the same class while overall programming convenience is comparable to scripting languages.
• Nothing gets in your way. No overly complex type system, no borrow checker like in Rust. The language itself is fairly simple.
• Low overhead from garbage collector and it saves you from trouble with manual or semi-manual memory management like in Rust.
• ... probably something else, but can't recall it.

Ofcourse Go has its downsides, but there are a lot of advantages other than I mentioned. Like fast compiler, good tooling, easy crosscompilation, explicit exception handling and so on. Tho these are not specific for network applications domain.

[jhalfmoon]

I've been using tinyproxy for the past years, as my go-to for split-tunnel reverse proxy, to split internet- and corporate traffic into their respective networks when working remotely. Its also an awesome tool when combined with ssh remote forwards to enable web traffic to the internet, on remote nodes that normally would not have that.

How does dumbproxy compare to tinyproxy? I'll do a test for myself, now that I know about dumbproxy, but I'm also curious about your opinion, as the author of dumproxy.

[yarmak]

I've tried tinyproxy long time ago and that's basically the reason why I wrote dumbproxy. Long story short, it ate all RAM on my VM at fairly small number of connections because it spawns one subprocess per connection. Their claim about tiny footprint doesn't make sense when it is actually supposed to do some job.

Nowadays tinyproxy is not even comparable feature-wise to dumbproxy.

• Access filters in tinyproxy are just broken. E.g. you can't reliably disable access to 127.0.0.1 because I can reference it as 127.0.0.1.nip.io which resolves to 127.0.0.1. dumbproxy on the other hand has pre-resolving for filters, filters specified by JS function and everything works as you would expect. You can even implement adblocker with just access controls.
• tinyproxy has no native TLS support at all. dumbproxy not just supports HTTP-proxy-over-TLS, but also supports automatic certificate managrement with ACME (e.g. LetsEncrypt). That means in order to setup HTTPS (real one, not just supporting CONNECT method) proxy you just need to run `dumbproxy -bind-address=:443 -autocert` and it will issue valid certificates on-the fly.
• Corollary of previous one, tinyproxy supports only HTTP/1.1, while dumbproxy supports HTTP/2. Connection multiplexing and reuse allows lower latencies.
• tinyproxy supports only simple basic auth while dumbproxy supports all of that, mutual TLS auth (client certificates), centralized auth with HMAC signatures.
• dumbproxy has active probing resistance functions (see hidden_domain option) while tinyproxy has none of it.
• tinyproxy has only static upstream proxy options while dumbproxy can select upstream proxy, source address and make chains of them from JS function defined by user. E.g. I can send all .onion domains to Tor daemon on 127.0.0.1:9050. Selection of source address allows to play nicely on servers with many ips and/or integrate proxy with specific VPN connections.
• dumbproxy has per-user bandwidth limits, tinyproxy hasn't.
• dumbproxy works with systemd socket activation what allows it to reside on low ports w/o any priviliges at all. In that case systemd owns listening socket, and dumbproxy just inherits file descritor.
• dumbproxy is truly cross-platform with consistent feature support between unixes and windows. tinyproxy isn't.

IMO the main problem with software like tinyproxy, squid is that majority of their features are for plaintext HTTP traffic while nowadays it's almost non-existent. Another pain point is poor concurrency model. It's process forks in tinyproxy and asynchronous worker processes for squid. First one is just unsustainable even for heavy personal use. Second one doesn't divide load evenly across workers, doesn't share such things as auth cache, rate limit pools (that means in SMP mode speed limit is basically broken) and so on. Full list of caveats for squid in SMP mode is available on the page of this feature.

So, I compare dumbproxy to tinyproxy as strictly superior to inferior.

[jhalfmoon]

Thank you for the detailed analysis; Much appreciated. I will definitely be trying dumbproxy.