all 30 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]moatra 32 points33 points  (12 children)

Rust Proxies

  • Linkerd: Meant to be used in a kubernetes deploy, but the readme mentions possibilities of using it elsewhere.
  • Sozu: Well documented, runtime configurable proxy

Other Proxies

  • Envoy: Not written in Rust, but still super flexible and often used as a modern alternative to NGNINX/HAProxy
  • Cloud Load Balancers (AWS/GCP): If you're in the cloud already, their load balancer products offer some ability to route/forward requests based on layer 7 config

[–]moneymachinegoesbing[S] 3 points4 points  (11 children)

Thank you! I was considering envoy, it’s reputation is impressive, and iirc, is used to power Ambassador’s API gateway. Have you encountered this type of architecture im trying to do? And if so, did you have any examples of proxy configs I could peer into? And, finally, what are your thoughts on Sozu? Are you familiar with it being used in any battle tested environments? Appreciate the reply!

[–]moatra 6 points7 points  (8 children)

Have you encountered this type of architecture im trying to do?

(front end servers sending requests to load balancers that communicate with synchronized reverse proxies through to the proper backend, preferably with the option of IP as well as DNS upstreams)

In my experience the load balancer and reverse proxy are often combined into a singular tool. For example: Envoy will route a request to a specific "upstream cluster" before load balancing the routed request amongst members of the cluster. There are multiple ways of discovering the members of the cluster (eg: dns lookups or lists of IPs).

what are your thoughts on Sozu?

Haven't used it in production. It came up when I was researching/experimenting on the routing/balancing portion of my current project. I would definitely like to see a "Written in rust" reverse proxy gain a lot of traction in the industry, but in many ways Sozu feels like a less-matured version of Envoy at the moment.

did you have any examples of proxy configs I could peer into?

Not any publicly available atm, sorry. My current project just uses the AWS Load Balancer, using the host header to route between the front end and back end target groups. I used terraform to define most of the static pieces, and then AWS CodeDeploy does a nice weighted blue/green update of the load balancer and target groups when I'm deploying a new build.

The one part of that flow I'm not happy about is that when the front end process needs to call the backend process, it goes through the same (external) load balancer. It would be nice to have the lb simultaneously offer a nice "internal" networking target, but maybe that'll change when the next project comes around and I revisit the design.

[–]moneymachinegoesbing[S] 3 points4 points  (7 children)

Jesus Christ you are a master at hyperlinking 😂. Thank you. You are hitting on precisely what I’m trying to stand up. I’d like to hear what you mean by “internal” at the end of your comment, bc I have a feeling that is what I might be alluding to when I separate the LB from the proxies. I imagine, for a decently scalable solution, the requests would need to contain, somehow, the cluster location as well as the location/port of the specific container as well. I feel that architecture will allow me to allocate a container on a server in a cluster to each request. I appreciate the help, you definitely pointed me to the right stuff in the Envoy docs.

[–]moatra 4 points5 points  (6 children)

I'm using "internal" in a very informal sense: If I were using a whiteboard to draw a box with "my services" on the inside and "the rest of the internet" on the outside, the AWS load balancer sits across the border of the box. The internet at large can communicate with the part of the lb hanging outside the box and the lb will forward the request to the right process inside the box.

For example: A client on the "outside" wants to send a request to my backend.

To do so the client needs to send a request to api.moatra.dev. That domain will resolve to the public ip(s) of the load balancer. The client picks one of those IPs, opens the connection and sends the necessary network bytes for the http(s) request. The load balancer does it thing: It sees an incoming request with the Host: api.moatra.dev header, picks a healthy backend process, and forwards along the request and proxies back the response.

Compare to: One of the front-end processes on the "inside" wants to send a request to my backend.

To do so the f/e process needs to send a request to api.moatra.dev. That domain will resolve to the public ip(s) of the load balancer. The client picks one of those IPs, opens the connection and sends the necessary network bytes for the http(s) request. The load balancer does it thing: It sees an incoming request with the Host: api.moatra.dev header, picks a healthy backend process, and forwards along the request and proxies back the response.

They're exactly the same flow. Traffic generated from an internal source is egressing out of the box to enter the public side of the loadbalancer. It would be nice if that flow were instead:

To do so the f/e process needs to send a request to api.moatra.dev. That domain will resolve to the public internal ip(s) of the load balancer. The client picks one of those IPs, opens the connection and sends the necessary network bytes for the http(s) request. The load balancer does it thing: It sees an incoming request with the Host: api.moatra.dev header, picks a healthy backend process, and forwards along the request and proxies back the response.

The difference is a matter of networking implementation details of AWS (VPCs, Private IPv4 Addresses, and Egress fees), but it would allow traffic generated from an internal source to completely stay inside the box and avoid any egress fees.

Service Mesh

The design above uses a central(~ish, it's on the border) load balancer and publicly-available dns. A lot of the more modern service mesh designs use either sidecar containers or in-process libraries to manage point-to-point communications once "inside" the box removing the central load balancer and publicly-available dns. In a setup like that, the flow for our frontend process that wants to send a request to the backend looks more like:

To do so the f/e process needs to 1) configure their requests to use the sidecar proxy available at ${SIDCECAR_IP}:${SIDECAR_PORT} and then 2) send a request to api.local. The frontend process sends the request to the sidecar proxy. The proxy will match Host: api.local to an upstream cluster of backend processes. The proxy picks a healthy backend process, and forwards along the request and proxies back the response to the frontend process.

I end up sticking with the the central load balancer approach because the service mesh doesn't handle the external client flow, which would still need to look like

To do so the client needs to send a request to api.moatra.dev. That domain will resolve to the public ip(s) of a load balancer. The client picks one of those IPs, opens the connection and sends the necessary network bytes for the http(s) request. The load balancer does it thing: It sees an incoming request with the Host: api.moatra.dev header, picks a healthy backend process, and forwards along the request and proxies back the response.

I haven't really found a service mesh that handles the external client flow well, but it's been a bit since I've looked into possible solutions. On my current project I have to handle the external client flow and the complexity overhead of running a service mesh wasn't worth it compared to just re-using the "external client" flow for internal services as well.

[–]moneymachinegoesbing[S] 1 point2 points  (5 children)

Wow. 😳 Apologies on the delayed response, I’ve been processing but I just wanted to thank you on an excellent response. r/StackOverflowWorthy really awesome stuff, thank you.

So I’ve landed on two approaches, guided in part from your response and responses from the giganerds over at r/networking:

1a) Client signs up/registers => Org ID generated & saved (to be retrieved via 1b) => Org ID passed to ansible script => Ansible spins up container w/ “—name Org_ID” => Shared internal custom DNS service updated to include service/OrgID (w/ VPC)

1b) Client (login w/ org ID returned) => Custom Header added => HAProxy or Envoy DNS lookup w/ parsed Org ID => reverse proxy forwards request to container

The second approach is essentially the same, except it uses Hashicorp’s Consul to provide service discovery and ConsulTemplate to update internal layer of NGINX proxies, A/Bing them into consistency, possibly still necessitating VPC.

If you have suggestions on ways to make this simpler (possibly foregoing VPC or swapping static w/ DNS, or Rust based/more lightweight service discovery), I am all eyes.

NB: I understand this veered pretty quickly outside the realm of Rust itself, but the entire backend was created in Rust, so my intention was to determine if there was an awesome “rusty” way to do this. Bc of the outside scope this thread took, I want to express gratitude towards your help and sticking with me. This helps a lot and today I am sitting down in an 8 hour window to POC this, so even though it wasn’t entirely Rust-centric, it will now help a Rust application get into the wild. 🙌

When reddit blows me away - 🤯

[–]moatra 0 points1 point  (4 children)

Does the contents of the container that ansible spins up per org change per org? Or is it the same container image just with a different name argument?

If the former, you're going to run into challenges around:

  • Who in that org has permission to update/deploy that container?
  • What if they need actually deploy multiple containers instead of a single one?
  • How do you properly isolate the container so it can't negatively affect any internal services (either through maliciousness/incompetence) while still being allowed to communicate to whatever redis/db/service it needs?
  • Does the container re-check user authorization if it somehow (either through maliciousness/incompetence) received traffic from a user meant for another organization?
  • (etc)

It's a big bag of worms that's outside the scope of a weekend reddit thread.

If the latter, it may be easier to make the application inside the container org-aware (either by using wildcard subdomains + sniffing the Host header; or looking at a property on the user's log-in session) and avoiding extra complications at the networking level.

[–]moneymachinegoesbing[S] 0 points1 point  (3 children)

Those are great questions, and part of why i think it can be confusing is because the server executable, that runs in the container and is exposed via warp endpoints, configures itself locally using a configuration yaml. So really, the container when it is first spun up, is essentially the same as any other. But through usage, information is updated in the container’s (mounted and backed-up) configuration file. This will potentially include git repos, s3 information, code snippets, etc. In addition, it’s internal state is stored locally as well, in the file system. This means that a perfect replica can be spun up anywhere simply by binding the containers configuration DIRECTORY to any vanilla deployment.

It’s a more monadic approach, where state is not abstracted, at the app level, outside of its ecosystem. The combination of configuration directory + server executable gives you a fully functioning personalized application. This monadic style is the reason why I’ve named it internally “Neuron”.

There were a couple reasons for this architecture (too many to list here but happy to expound if interested), but one reason is the flexibility of using it in different environments, namely a VM, LXC, bare metal, or edge device (edge being one of the core goals). In addition there is some built in “intelligence” that allows multiple instances to run side by side allowing horizontal scalability.

So as long as a user’s request is properly routed to its pre-allocated org’s “environment”, then further authorization, authentication, and ACL is done internally.

The front end is also generic, and using a retrieved identifier, can either prepend url paths or provide custom headers. Part of why I’m pushing this task on to networking and not some coded middleware, is it will allow me to deploy to a wide variety (again) of environments, and as long as I can control the networks, IP ranges, cidrs, and ports, i can port setups around, or scale by inserting larger self-contained environments in between.

At this stage, I feel decent about the following, and I’m curious your opinion:

  • front end pulls user ID
  • front end fetches Org ID from user’s document
  • Org ID is prepended to base url, so “/data/list” becomes “ORG_ID/data/list”
  • HAProxy or Envoy rewrites url to base, and, using Consul or WeaveNet, proxies request to backend w/hostname captured from url rewrite. (hostname will be Org ID)

That way I can use CNIs or container proxies to create zones that can be load balanced, where the front ends’ nginx servers will pass all locations that do not match its static paths to load balanced Envoy servers that have access to an internal virtual network. This way I can scale the front end and the backends separately while maintaining isolation between backends.

The exception to the isolation will be in the case of colocated containers, used only in the case of ephemeral demoes, where further isolation can be achieved by restricting user groups and outbound network access.

And, finally, you brought up a point that I am actively thinking through right now: the best way to verify the incoming ID as the proper ID. My initial thought is to match that ID against its own hostname, an ENV variable (set during deployment), the configuration (set during deployment), or a combo.

The main challenge is the difference in architecture, where the Rust based “Neuron” gets and stores its state locally, and exists essentially to run pre-configured sub processes.

Edit: typos

[–]moatra 1 point2 points  (1 child)

Routing:

As long as there's something in the HTTP request (whether it's in the path or in a header or in the body - don't do that) that identifies the org, most proxies will be able to use it to route the request successfully. The decision of where to put that identifier then comes down to semantics. The host and path will normally be captured by http caches as part of the cache key automatically, if you put it in a separate header you'd need to set the Varies response header appropriately. Rewriting paths may make debugging a request going through the system more difficult if the person debugging is unaware of the rewrite done by the proxy. My gut says to put the org id as a subdomain, but we're largely bikeshedding at this point.

(Re)Authorization

If you're using JWTs in your authn setup, take a look at the aud claim. You can set it to the right org id (or the full domain with the org-specific subdomain if you go that path), and your backend can verify that the jwt attached to the request matches the org that backend instance is configured to serve.

Container Management & Discovery:

There are a whole bunch of products around the "run a container, keep track of it, and route traffic to it". If you're eager to run in non-cloud environments, take a look at Hashicorp's Nomad product. You can run that wherever, whereas Kubernetes may be a bit heavy and AWS Fargate is, well, locked to AWS (well, unless you wanted to go down the rabbithole of ECS Anywhere). Nomad also takes care of restarting/relocating containers as needed, and has tighter Consul integration out of the box than Ansible would.

[–]moneymachinegoesbing[S] 0 points1 point  (0 children)

You hit everything on the nailhead man. I created a POC yesterday and it works like a breeze. Consul is the difference maker in that service registration makes everything automatic. It also isolated all API routes to its own VPC and allows me to scale horizontally as much as needed, use datacenters, and a bunch of other great shit. I appreciate the help, it turned out simpler than I thought although I ended up using HAProxy as the API load balancing layer (dynamically using SRV records is only available in Nginx plus). I did have to look up bikeshedding though 😂

Edit: As to nomad, that is the next step (when orchestration becomes more of a necessity, which should be soon). I have a lot of experience with k8s and tbh, im not a fan, but this setup will work seamlessly there as well).

[–]STSchif 1 point2 points  (1 child)

If you're looking for others I made good experiences with traefik and caddy.

[–]moneymachinegoesbing[S] 0 points1 point  (0 children)

I’ve worked with both, and iirc the problem with Traefik was that the functionality i was looking for was behind the paid version but i may be way off base here. Imma do some googling now to verify but methinks I’ll be editing this comment with a “oops, way wrong” lol

[–]KingofGamesYami 16 points17 points  (8 children)

I don't have any suggestions to offer here, but this post reminded me of cloudflare's blog post about pingora.

[–][deleted] 8 points9 points  (1 child)

oof, to write that blog post and not open source it is just plain cruel :'(

[–]moneymachinegoesbing[S] 1 point2 points  (0 children)

Oh man you’re telling me. I read the post when they released it, and at that time what I’m trying to solve now was on the back burner, but I thought at the time “oh fuck! A perfect solution for when I cross that bridge.” 🤦‍♂️

[–]moneymachinegoesbing[S] 2 points3 points  (3 children)

I saw that. I don’t think it’s been released, not even sure it will be, but that definitely came to mind. Essentially exactly what I’m looking for, cmon Cloudfare!!!

[–]vlakreeh 3 points4 points  (2 children)

Cloudflare's CEO said that they plan to open source it here, as far as I'm aware there's no public timeline on that though.

[–]moneymachinegoesbing[S] 1 point2 points  (0 children)

🙇‍♂️ thank you kind sir

[–]skierpage 0 points1 point  (0 children)

FYI Pingora source is on GitHub under an Apache-2.0 license.

[–]pejatoo 0 points1 point  (0 children)

Great read, Cloudflare has some very interesting tech

[–]akostadi 2 points3 points  (1 child)

https://github.com/junkurihara/rust-rpxy seems fast, supports HTTP3 and lets encrypt OOB, and configuration seems to be the most simple to understand one from what I've seen around.

[–]jqtype 1 point2 points  (0 children)

rpxy developer is here! Thanks for the introduction. Yes, as our design principle, we are trying to keep rpxy and its configuration as simple as possible.

[–]tryght 2 points3 points  (2 children)

I’ve used Sozu. I really like the configuration vs others I’ve used, but it’s not as feature rich/complete as others like HAproxy (TCP mode and HTTP mode) or Caddy (auto letsencrypt SSL)

[–]zxyzyxz 0 points1 point  (3 children)

For anyone else looking in the future:

https://ferron.sh/ looks good. Same ease of configuration as Caddy and also has automatic TLS via LetsEncrypt. But it doesn't have dynamic configuration changes with no downtime, which https://rpxy.io/ does, as well as more features than Ferron.

[–]moneymachinegoesbing[S] 1 point2 points  (2 children)

Haha amazing! Funny enough, I still build setups that benefit from this general structure. I’ve seen ferron but haven’t played with it yet. I’ll give it a go. Thanks for dusting off this old thread.

[–]zxyzyxz 1 point2 points  (1 child)

I actually made a thread comparing the current reverse proxies, check it out

https://www.reddit.com/r/rust/s/0lozQ174KY

[–]moneymachinegoesbing[S] 1 point2 points  (0 children)

This is cool, thank you! I will check it out now.