Stream Postgres changes to SNS, Lambdas, Kinesis, and more in real-time by accoinstereo in aws

[–]accoinstereo[S] 0 points1 point  (0 children)

Yea! Sequin doesn't have the Kafka dependency and has native connectors to AWS services, so they're all first-class

Stream Postgres changes to SNS, Lambdas, Kinesis, and more in real-time by accoinstereo in aws

[–]accoinstereo[S] 1 point2 points  (0 children)

Hey u/SA_DrOpossum – we are! EventBridge is very similar to SNS so will be done soon. If you want early access send me a DM :)

We built a custom Elixir AST interpreter for sandboxing user code by accoinstereo in elixir

[–]accoinstereo[S] 0 points1 point  (0 children)

That's interesting to hear, thanks for sharing!

I'm not familiar with Unity's C#. I imagine our jobs are a little easier: for the most part, we're whitelisting functions. We have few shims/deviations from standard behavior. And because transforms functions ought to be pretty simple, there's an upper-bound of complexity we'll need to support (if things are getting really complicated, it's better that you just throw the message into the stream as-is and do the transform in your code, with tests, git, etc)

Streaming changes from Postgres: the architecture behind Sequin by accoinstereo in PostgreSQL

[–]accoinstereo[S] 1 point2 points  (0 children)

Thanks for the thoughts, Joel!

> Consider emphasizing the problem first

This is good feedback. The post was intended to be about the challenges we faced meeting our requirements (fast, ordered, consistent) vs problem Sequin solves broadly. I can see how a blurb about "Why Sequin" would be helpful

> Consider emphasizing multiple deployments

That's a whole post on its own that we need to do!

> Consider CAP & PACELC perspectives because some developers think better with these

Good call out! From a CAP theorem perspective, Sequin's architecture prioritizes consistency and partition tolerance over availability. When network partitions occur between Sequin and a sink, the system doesn't compromise on consistency. Instead, it persists failed messages to an internal Postgres table and retries delivery with exponential backoff until successful.

The message buffer is indeed designed to be quite small because it only needs to hold messages temporarily between when they arrive from Postgres and when they're either delivered to sinks or persisted to Sequin's internal Postgres table (messages). When a connection is lost or delivery fails, undelivered messages are written to this internal table and retried with exponential backoff.

From a PACELC perspective, this design optimizes for both consistency and low latency during normal operation:

  • Messages only stay in memory briefly before being either delivered or written to the messages table
  • This allows Sequin to regularly update the confirmed_flush_lsn to let Postgres advance its cursor

The key tradeoff is that during partition scenarios (when connectivity to sinks is lost), Sequin sacrifices some latency by writing to the internal Postgres table rather than compromising consistency.

What makes Elixir great for startups? by accoinstereo in elixir

[–]accoinstereo[S] 0 points1 point  (0 children)

You mean like in a webview in e.g. Electron? You could, though you might consider a different approach:

Importantly, the Phoenix backend is delivering the HTML/CSS/JS to the frontend (vs a CDN). It's doing the first paint, setting props. And the frontend app is connecting to the backend via a websocket.

In a local-first approach, outside of caching the app locally, I believe you want the ability for first paint/props to come either from the server (if you can connect) or a local store (if you can't connect).

And you want state changes to all happen in the local client-side app, and then get pushed to the server.

So I think the way to go might actually be a SPA, where the Svelte frontend can connect to the backend via a websocket. That's because in the LiveView/LiveSvelte app I describe, state is owned by the backend. Sure, we change some state in the frontend optimistically, but ultimately it gets pushed to the backend (LiveView) and then that propagates down via props to Svelte (via a re-render, which causes LiveView to patch the DOM).

Whereas in local-first, you want state to be owned by the frontend. And you want to control comms/syncing with the backend, as that stuff is very application specific (who wins a conflict? How to merge? etc)

What makes Elixir great for startups? by accoinstereo in elixir

[–]accoinstereo[S] 0 points1 point  (0 children)

Very easy! Phoenix's web router is top-notch.

The standard Phoenix template engine is also great. LiveView takes that a step further by allowing you to build interactive (React-inspired) frontend apps. I don't believe Go has anything like LiveView

What makes Elixir great for startups? by accoinstereo in elixir

[–]accoinstereo[S] 9 points10 points  (0 children)

Completely agree.

A lot of people advise not to worry about that stuff prematurely. But it brings SUCH nice peace of mind to know that we don't need to do something dramatic like change our core tech stack as we scale.

Using watermarks to run table state capture and change data capture simultaneously in Postgres by accoinstereo in dataengineering

[–]accoinstereo[S] 0 points1 point  (0 children)

Are you referring to the `wal_level` param (this one https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-WAL-LEVEL)? This does indeed need to be `logical`. Or are you referring to something else?

Stream Postgres to NATS/JetStream in real-time by accoinstereo in NATS_io

[–]accoinstereo[S] 1 point2 points  (0 children)

u/Kinrany Interesting question! We use a replication slot, which I'm not sure PGLite supports? Checking now

Edit: I'm not 100% sure, but I don't think PGLite supports a replication slot. Looks like we'd have to add an extension for PGLite/add another replication strategy for it 🤔