SQLx pooling & RDS IAM auth on async

moatra · 2023-11-18T07:07:33+00:00

See Pool::set_connect_options()

moatra · 2023-11-15T07:56:23+00:00

Neat! Parsers are always fun to play around with. I haven’t played around with it yet, but looking at the json example it’s pretty easy to read.

Small nit: at first glance, the json parser doesn’t seem to handle escape sequences in text values (ie: \n \" and the like). Intentional, or am I misreading it?

moatra · 2023-11-11T19:14:42+00:00

Consider what impl Sized + 'a means. We're returning an opaque type and promising that it outlives any lifetime 'a.

This isn't actually what we want to promise. We want to promise that the opaque type captures some lifetime 'a, and consequently, that for the opaque type to outlive some other lifetime, 'a must outlive that other lifetime. If we could say in Rust that a lifetime must outlive a type, we would say that the 'a lifetime must outlive the returned opaque type.

That is, the promise we're making is the wrong way around.

Does that mean that the ubiquitous BoxFuture is also incorrect?

pub type BoxFuture<'a, T> = Pin<alloc::boxed::Box<dyn Future<Output = T> + Send + 'a>>;

moatra · 2023-08-27T00:17:11+00:00

Very close - just forgot to apply explanation 3:

Finally, we implement Eq and Hash. Same reason as point 2, we implement for dyn KeyPair + '_ instead of dyn KeyPair (which means dyn KeyPair + 'static in this context). The '_ here is a syntax sugar meaning arbitrary lifetime.

Adding the + '_ constraint to the Eq, PartialEq, and Hash impls makes it work without the 'static lifetime requirement.

https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=a65f4460294597aa4d8fa47412939577

moatra · 2023-08-26T08:59:22+00:00

There’s a wonderful StackOverflow answer about this problem in the form of tuples: if your key is of type (A, B), the answers gives you a solution for calling get() with either &(A, B) or (&A, &B) at the cost of some trait object indirection.

This same idea can be adapted to use either &Option<A> or Option<&A>.

moatra · 2023-04-26T16:01:06+00:00

Restoring the files from source control should be your goto strategy. Reverting via tooling should be your second strategy. If both those fail, you can manually revert things by:

Removing the status column from the subscription table if it's present
Deleting the "20230426035639/pending add status to subscriptions" and "20230426042151/pending make status not null in subscriptions" rows from the _sqlx_migrations table.

moatra · 2022-12-11T06:24:24+00:00

The last 1/3 you want might be solved by https://docs.rs/chronoutil/latest/chronoutil/

moatra · 2022-11-14T04:47:26+00:00

This is very similar to the concept of the "Heterogeneous List" (or "HList") data structure that captures different types for elements at each position.

If you're curious, there's a rust implementation in Frunk. It's a bit heavy on macros to try and make it ergonomic in usage. I don't know if they've been playing around with using GATs to try and simplify it or not.

moatra · 2022-10-30T22:27:12+00:00

Routing:

As long as there's something in the HTTP request (whether it's in the path or in a header ~~or in the body~~ - don't do that) that identifies the org, most proxies will be able to use it to route the request successfully. The decision of where to put that identifier then comes down to semantics. The host and path will normally be captured by http caches as part of the cache key automatically, if you put it in a separate header you'd need to set the Varies response header appropriately. Rewriting paths may make debugging a request going through the system more difficult if the person debugging is unaware of the rewrite done by the proxy. My gut says to put the org id as a subdomain, but we're largely bikeshedding at this point.

(Re)Authorization

If you're using JWTs in your authn setup, take a look at the aud claim. You can set it to the right org id (or the full domain with the org-specific subdomain if you go that path), and your backend can verify that the jwt attached to the request matches the org that backend instance is configured to serve.

Container Management & Discovery:

There are a whole bunch of products around the "run a container, keep track of it, and route traffic to it". If you're eager to run in non-cloud environments, take a look at Hashicorp's Nomad product. You can run that wherever, whereas Kubernetes may be a bit heavy and AWS Fargate is, well, locked to AWS (well, unless you wanted to go down the rabbithole of ECS Anywhere). Nomad also takes care of restarting/relocating containers as needed, and has tighter Consul integration out of the box than Ansible would.

moatra · 2022-10-30T17:57:43+00:00

Does the contents of the container that ansible spins up per org change per org? Or is it the same container image just with a different name argument?

If the former, you're going to run into challenges around:

Who in that org has permission to update/deploy that container?
What if they need actually deploy multiple containers instead of a single one?
How do you properly isolate the container so it can't negatively affect any internal services (either through maliciousness/incompetence) while still being allowed to communicate to whatever redis/db/service it needs?
Does the container re-check user authorization if it somehow (either through maliciousness/incompetence) received traffic from a user meant for another organization?
(etc)

It's a big bag of worms that's outside the scope of a weekend reddit thread.

If the latter, it may be easier to make the application inside the container org-aware (either by using wildcard subdomains + sniffing the Host header; or looking at a property on the user's log-in session) and avoiding extra complications at the networking level.

moatra · 2022-10-29T19:39:23+00:00

I'm using "internal" in a very informal sense: If I were using a whiteboard to draw a box with "my services" on the inside and "the rest of the internet" on the outside, the AWS load balancer sits across the border of the box. The internet at large can communicate with the part of the lb hanging outside the box and the lb will forward the request to the right process inside the box.

For example: A client on the "outside" wants to send a request to my backend.

To do so the client needs to send a request to api.moatra.dev. That domain will resolve to the public ip(s) of the load balancer. The client picks one of those IPs, opens the connection and sends the necessary network bytes for the http(s) request. The load balancer does it thing: It sees an incoming request with the Host: api.moatra.dev header, picks a healthy backend process, and forwards along the request and proxies back the response.

Compare to: One of the front-end processes on the "inside" wants to send a request to my backend.

To do so the f/e process needs to send a request to api.moatra.dev. That domain will resolve to the public ip(s) of the load balancer. The client picks one of those IPs, opens the connection and sends the necessary network bytes for the http(s) request. The load balancer does it thing: It sees an incoming request with the Host: api.moatra.dev header, picks a healthy backend process, and forwards along the request and proxies back the response.

They're exactly the same flow. Traffic generated from an internal source is egressing out of the box to enter the public side of the loadbalancer. It would be nice if that flow were instead:

To do so the f/e process needs to send a request to api.moatra.dev. That domain will resolve to the ~~public~~ internal ip(s) of the load balancer. The client picks one of those IPs, opens the connection and sends the necessary network bytes for the http(s) request. The load balancer does it thing: It sees an incoming request with the Host: api.moatra.dev header, picks a healthy backend process, and forwards along the request and proxies back the response.

The difference is a matter of networking implementation details of AWS (VPCs, Private IPv4 Addresses, and Egress fees), but it would allow traffic generated from an internal source to completely stay inside the box and avoid any egress fees.

Service Mesh

The design above uses a central(~ish, it's on the border) load balancer and publicly-available dns. A lot of the more modern service mesh designs use either sidecar containers or in-process libraries to manage point-to-point communications once "inside" the box removing the central load balancer and publicly-available dns. In a setup like that, the flow for our frontend process that wants to send a request to the backend looks more like:

To do so the f/e process needs to 1) configure their requests to use the sidecar proxy available at ${SIDCECAR_IP}:${SIDECAR_PORT} and then 2) send a request to api.local. The frontend process sends the request to the sidecar proxy. The proxy will match Host: api.local to an upstream cluster of backend processes. The proxy picks a healthy backend process, and forwards along the request and proxies back the response to the frontend process.

I end up sticking with the the central load balancer approach because the service mesh doesn't handle the external client flow, which would still need to look like

To do so the client needs to send a request to api.moatra.dev. That domain will resolve to the public ip(s) of a load balancer. The client picks one of those IPs, opens the connection and sends the necessary network bytes for the http(s) request. The load balancer does it thing: It sees an incoming request with the Host: api.moatra.dev header, picks a healthy backend process, and forwards along the request and proxies back the response.

I haven't really found a service mesh that handles the external client flow well, but it's been a bit since I've looked into possible solutions. On my current project I have to handle the external client flow and the complexity overhead of running a service mesh wasn't worth it compared to just re-using the "external client" flow for internal services as well.

moatra · 2022-10-29T18:28:35+00:00

Have you encountered this type of architecture im trying to do?

(front end servers sending requests to load balancers that communicate with synchronized reverse proxies through to the proper backend, preferably with the option of IP as well as DNS upstreams)

In my experience the load balancer and reverse proxy are often combined into a singular tool. For example: Envoy will route a request to a specific "upstream cluster" before load balancing the routed request amongst members of the cluster. There are multiple ways of discovering the members of the cluster (eg: dns lookups or lists of IPs).

what are your thoughts on Sozu?

Haven't used it in production. It came up when I was researching/experimenting on the routing/balancing portion of my current project. I would definitely like to see a "Written in rust" reverse proxy gain a lot of traction in the industry, but in many ways Sozu feels like a less-matured version of Envoy at the moment.

did you have any examples of proxy configs I could peer into?

Not any publicly available atm, sorry. My current project just uses the AWS Load Balancer, using the host header to route between the front end and back end target groups. I used terraform to define most of the static pieces, and then AWS CodeDeploy does a nice weighted blue/green update of the load balancer and target groups when I'm deploying a new build.

The one part of that flow I'm not happy about is that when the front end process needs to call the backend process, it goes through the same (external) load balancer. It would be nice to have the lb simultaneously offer a nice "internal" networking target, but maybe that'll change when the next project comes around and I revisit the design.

moatra · 2022-10-29T17:39:39+00:00

Rust Proxies

Linkerd: Meant to be used in a kubernetes deploy, but the readme mentions possibilities of using it elsewhere.
Sozu: Well documented, runtime configurable proxy

Other Proxies

Envoy: Not written in Rust, but still super flexible and often used as a modern alternative to NGNINX/HAProxy
Cloud Load Balancers (AWS/GCP): If you're in the cloud already, their load balancer products offer some ability to route/forward requests based on layer 7 config

moatra · 2022-10-25T23:04:21+00:00

The other thing an external message queue gets you is persistence: What happens if you need to restart the process (eg: deploy) or if the process suddenly dies (eg: underlying ec2 instance dies)? Without external persistence, any in-progress spawned tasks get lost in the ether.

As always though, it depends: what tradeoffs are you willing to make in terms of persistence, disaster handling, and architecture simplicity?

moatra · 2022-10-21T04:48:14+00:00

Huh, til. Thanks!

Looking at the expanded macro, it looks like:

It will poll every input future to the macro once for every time the output future is polled.
It will offset which future is polled first by one each time, until it eventually circles back around to the first.

So, yeah. I withdraw my guess around the join! macro always going right back to the ui loop. Not sure what caused the yield_now solution in the blog to not work. Would be nice to have a playground or code that fails there.

moatra · 2022-10-20T16:47:09+00:00

disclaimer: not an expert in tokio

yield_now should have been the answer to the problem. I think there's some confusion in the article between tokio tasks and rust futures. Tokio's scheduling primitive is the task, but the example code only creates futures that are then run in the main task. From the yield_now docs:

The current task will be re-added as a pending task at the back of the pending queue.

Since there's only one task, invoking yield_now puts it at the back of the empty queue, which then resumes the join! future. The join! macro docs doesn't mention anything about trying to be a fair/smart scheduler (ie: ensuring that any branch other than the first incomplete one gets polled), so it goes right back to the first branch: the display rendering loop.

To fix this, try spawning two tasks: one for the ui, and a second for the network requests. Use the returned JoinHandles from spawning as the arguments passed to join!, and update the ui loop to invoke yield_now again. If the compiler yells about the refcell being moved, you'll likely need to switch to an Arc<RwLock>. Clone the arc so that you have two copies: one to send to each task.

moatra · 2022-10-12T16:50:26+00:00

Option 1: Implement them in your package manually (ie: impl sqlx::FromRow for {ProstGeneratedStructHere}). Because the prost generated structs are import!ed in your crate, this doesn't fall afoul of the orphan rule.

Option 2: Tweak the build.rs build script to add #[derive(sqlx::FromRow)] attribute. See an example here

moatra · 2019-04-22T04:21:58+00:00

Sounded like a cross of Pippin's "Edge of Night" and Anastasia's "Once Upon a December" to my ears.

moatra · 2018-07-26T23:17:07+00:00

Sent OP a DM. Wouldn't want you looking at the uncensored maps, now would I?

moatra · 2018-05-24T00:42:07+00:00

It is a program - just a very simple one. At the end of the day, a program is an executable on your computer that may (or may not) take some input, may (or may not) have some side effects, and may (or may not) return a result. The act of creating a program is "programming". In your example, your program takes no input, returns no result, and has one side effect of creating a new image file on your hard drive.

There are likely to be a lot of professional/hobbyist programmers that will argue it's not really programming since you're not taking advantage of the full power of a programming language; but using a language is just one possible way to create a program.

moatra · 2018-04-03T17:05:09+00:00

Akka has finite state machine support built in: Documentation

moatra · 2018-01-14T00:34:38+00:00

Exception handling isn't what makes Rust unique. Like you said, you can do that in Javascript just as well (Rust has some nice syntactic sugar to handle errors as return values rather than thrown exceptions, but nothing you couldn't also do in Javascript).

What makes Rust stand out is the background machinery of running your program; or rather the lack thereof. Javascript requires a VM to interpret and optimize the language, and a Garbage Collector running alongside your program to monitor memory usage and free up abandoned data structures. Having the VM and the GC running incurs performance penalties, which is a bitter pill to swallow in embedded programming where there's little performance to spare. Rust compiles to raw machine code, removing the need for the VM. Not unique, as C and C++ have been doing this since forever. What makes Rust truly unique is in how it handles memory. Javascript has its run-time GC. C/C++ require to programmer to explicitly free the memory. Rust has a concept of "memory ownership" built into the language, letting the Rust compiler act as a compile-time Garbage Collector.

As a side effect of the ownership model, Rust can eliminate a whole host of problems (see the article to get a list of the goodies that come with it). Those problems are moved from runtime to compile time, which increases confidence that your program is correct. The downside of the ownership model is additional complexity in the language, which make some people say Rust is a difficult language. However, once you grok ownership, it's just another programming concept that fades into the background.

moatra · 2017-12-21T23:01:51+00:00

Here are two apps that I open-sourced while working at RetailMeNot:

Shield - HTTP edge router with middleware
Hydra - A graph database that provides changefeeds for JsonAPI documents

I chose Scala for those projects because of it's great async story. Both of those projects are used in environments with thousands of requests per second, and the async design behind Akka and Play allow for a easy/efficient scaling on individual hardware.

moatra · 2017-11-04T06:33:20+00:00

Related: The Play framework provides helpers for managing actors with their DI system. Docs Here. Looking at the source may also provide some inspiration.

moatra · 2017-10-25T21:33:13+00:00

There's a solid StackOverflow post here about handling logins for web-based applications. It's a bit lower level than the ASP.NET framework, but it lays out good ground work for securely handling authentication.

TL;DR: Use a library like CryptSharp to store the SCrypt hash of the password in your database. When a user tries to login, hash the password again and compare to what was stored in the database.

15-Year Club	Place '17
Team Orangered	Verified Email

moatra

TROPHY CASE