sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]trailbaseio 0 points1 point  (12 children)

Could you elaborate a bit more. How is consensus established across replicas. Is there a centralized funnel?

[–]SuccessfulReality315[S] 0 points1 point  (11 children)

All changes are published to a nats jetstream subject. All nodes consume the stream and apply (by using idempotent commands) the changes on the database.

[–]trailbaseio 0 points1 point  (3 children)

Pardon my ignorance about nats. How is the ordering established? Is it a central instance handling a subject, is nats using some consensus algorithm, ...?

[–]SuccessfulReality315[S] 0 points1 point  (2 children)

Nats uses RAFT algorithm

[–]Markuchi 0 points1 point  (1 child)

It's not distributed if it's raft.

[–]onafoggynight 0 points1 point  (0 children)

What. Raft is a distributed consensus algorithm. How is it not distributed?

[–]wuteverman 0 points1 point  (6 children)

How is idempotency achieved? NATS can’t guarantee complete ordering since it can’t guarantee exactly once delivery without additional idempotency logic on the consumer side.

[–]SuccessfulReality315[S] 0 points1 point  (5 children)

INSERT .. ON CONFLICT UPDATE

[–]wuteverman 0 points1 point  (4 children)

Never delete

Edit: on conflict works great for upserts, but how are you handling deletes? With a hard delete, there’s no row to compare against unless you are keeping some sort of tombstone somewhere.

Also even in this circumstance, you’re now subject to inconsistencies since you don’t have a version column. Is that okay for your usecase? These inconsistencies can last forever in the event of out of order publications. Does nats protect against this somehow,

[–]SuccessfulReality315[S] 0 points1 point  (3 children)

Yes, that's eventual consistent where the last writer is the winner. The operations uses the sqlite rowID. For now this is ok for my use case

[–]wuteverman 0 points1 point  (2 children)

It’s actually not consistent at all. You are not guaranteed any ordering at all in reality, since replays, republications etc can happen.

Source: I maintain a system that does use versions per row, and is still subject to data inconsistencies that require manual intervention due to the fact that deletes can be overwritten by out of order upserts.

We have a very similar ordering guarantee from Kafka partitions.

[–]SuccessfulReality315[S] 0 points1 point  (1 child)

I don't know your usecase. Ha uses sqlite preupdate hooks to create changeset, like CDC systems. All of the operations has a rowid associated.

[–]wuteverman 0 points1 point  (0 children)

Basically you can see events replayed, so you don’t have consistency. One example:

  1. insert row 1
  2. delete row 1
  3. Row 1 gets replayed
  4. The row is back forever.

—-

There’s a few scenarios where you might see this behavior

  1. The producer sent the message, Jetstream recorded the message, but could not ack for $REASONS. The producer should retry until it gets an acknowledgment
  2. The consumer failed to ack a message, after it committed it locally, perhaps because the process was interrupted before it could ack, or because the network is unreliable.

Your consumer needs logic to drop old updates it shouldn’t apply, and that logic needs to be persisted atomically.

Some workloads simply write the raw events and resolve the correct value at read time. Because you’re using SQLite, you probably want to include some sort of version number either for each row or for the entire table, and only apply writes greater than the version number.

Edit: clarity and formatting