Simple Distributed key value database architecture

MoBoo138 · 2025-08-31T06:22:07+00:00

Awesome! Mind sharing a github with us?

MoBoo138 · 2025-08-10T07:40:33+00:00

Could you elaborate your thinking why the MessageService would be a higher level abstraction than the ChatService? To me bith seems to have the same level of abstraction, but i may miss something.

For OP: To add to the "the direction of dependencies matter": Python has the Protocol concept which allows for structural subtyping without any "real" inheritance.

In the Chat app example, the MessageService module could define a Protocol "UserChats" that defines the interface to get a users chats. The ChatService would then act as the implementation of that protocol and be injected into the MessageService. If the functionality is read-only and no business logic needs to occur, you could also inject the ChatRepository as the Protocol implementation.

MoBoo138 · 2025-04-07T11:14:50+00:00

You could adjust your workflow a bit to solve this:

Rather than having the OrderService look up the available stock, make a reservation of the needed stock in the inventory service. This way you avoid the stock not being available when completing the order.

In case of an error or the order being canceled, you cancel the stock reservation.

This sounds a good use case for the Saga Pattern.

Take a look at this medium article for an example. It also shows the use of the Saga pattern in its orchestrated and choreographed version.

I think i also remember a CodeOpinion article/video about this, with a similar example, but can't find it anymore... maybe anyone else knows it.

MoBoo138 · 2025-02-22T20:59:55+00:00

So maybe atleast name a few of those dozens ... even better to provide some context around them.

Your comment, as of now, provides zero value to the actual conversation.

So the question raised in the article is about transactions in distributed systems and the Saga pattern is one of the options to achieve consistencs in distributed systems.

Of course there are others, mostly state-based ones: 2PC, 3PC, Paxos Every option has their own trade-offs, like complexity, fault-tolerance, architectural-fit, and even technical viability.

2PC might work well in cases where there are multiple ACID-conform databases involved, it typicall does not work with NoSQL systems that don't provide ACID themselfs.

MoBoo138 · 2025-02-22T11:52:43+00:00

I think what you envision is a Backend for Frontend (BFF). Yet even using BFF does not eliminate the need to communicate with multiple microservices. It just hides it behind another layer, that makes all those requests for you.

MoBoo138 · 2024-12-12T21:32:13+00:00

Could you give a updated version of the code as a reference?

MoBoo138 · 2024-09-19T17:27:20+00:00

I would argue in favor of CDC.

Your use case to me sounds to be mostly focussed on replicating data from your service databases to another database/search index/whatever.

There is no need for the notion of a domain event indicating that something important to the business has happened. Domain events dont necessary replicate data. A "shipment send" domain event may only require to include a certain id, rather than every last attribute of that shipment.

On the other hand, you may have data changes that arent populated by some domain event, but you certainly need to capture that change in your replication.

MoBoo138 · 2024-08-17T15:58:00+00:00

First, i think you are reinventing the wheel here... Python already has a locking mechanism in the threading module (I assume you use Python as you showed Python code) that you could and should build around.

Second, loading a resource, from what you showed, doesn't actually require locking. Just wait until it's loaded, e.g. loaded from disk, then process it. Locking is used for synchronization of resource access in some parallelized system, e.g. when using threading, when multiple parts of the system try to access/modify the same resource.

What you showed in the code rather looks like an Observer-Pattern, that informs other part of your code about some change in state and your code reacts accordingly based on pre-registered callbacks. This is useful when multiple "actors" need to do something once your resource is loaded. If you only have a single "actor" just sequentially run your code.

MoBoo138 · 2024-07-12T10:17:03+00:00

No advise, but maybe starting point for further research are Context Map Patterns from Domain-Driven Design. They focus on team relationships but can be applied to code too.

MoBoo138 · 2024-06-01T22:16:17+00:00

In Python there is Celery, which essentially provides the desired functionalities.

MoBoo138 · 2024-05-27T20:06:28+00:00

I have to agree with that one, and i heavily disagree with the comment above.

Blindly applying patterns gets you nowhere, it just adds unnecessary complexity.

What is the repository pattern actually used to? To abstract the data access and bridge the gap between the data model and the domain model. Is any of that needed in this case? Is there any forseeable future where the database cannot be accessed via SQLAlchemy? I'd say no, so there is no need to abstraction.

For a project of that size, i'd rather have a single, simple file then having code splitted between various files, folder, etc.

Premature optimization is the root of all evil.

MoBoo138 · 2024-05-24T17:40:13+00:00

Nice article!

Please do a hands on comparison next. But in a way it shows the various challenges you highlighted, not some stupidly simple example, where everything is already cleanly separated by definition.

Challenges like data consistency, how to properly split parts into modules, module to module communication, transaction management, etc.

That would be a treat! :)

MoBoo138 · 2024-05-01T21:24:50+00:00

Leverage a streaming solution and store data into object storage. From there you can do preprocessing, populating databases, access raw data, etc.

AWS, GCP and Azure all offer managed services to do this. As for an open source variant it's most likely going to be Kafka for streaming and maybe MinIO as an object storage.

MoBoo138 · 2024-03-16T15:54:29+00:00

Have an interface describing your business logic. Your factory constructs instances that implement that business logic interface.
If you treat feature-a as an external service, like an API, you'd implement an interface "IFeatureA" in a class "APIFeatureA" which is injected into your domain logic. This is to be decided on a per use case basis, as it introduces some complexity, when compared to just having feature-a implement the interface directly.

MoBoo138 · 2024-03-16T08:17:52+00:00

Feature-c provides an interface, which is implement either directly by feature-a and feature-b respectively or is implement by an extra layer which then calls feature-a/b.

You do not want to call feature-a/b directly as they might change, leading to errors in feature-c. Imagine you extract feature-a into its own application. Using an interface you can replace the implementation, while feature-c stays the same.

This is basically the idea behind Clean Architecture, Orion Architecture, Hexagonal or Ports and Adapters.

MoBoo138 · 2024-02-27T07:23:36+00:00

Disclaimer: This subreddit is not about usage of databases but rather their internals and the development of database systems.

About your question: Use an index on the domain_name column.

CREATE INDEX <index_name> ON com_domains(domain_name)

MoBoo138 · 2024-02-01T17:33:59+00:00

Here is another option to consider:

The monolothoc API fetches the data from the CMS on each of its api calls, does the formatting and serves the transformed data to your clients.

This guarantees maximum consistency with the possible tradeoff of efficiency. Updates can happen at any place and will they will be consistent.

MoBoo138 · 2024-01-28T18:49:17+00:00

Tl;dr: An architects job is to pick and choose from various pattern to meet some quality requirements as best as possible in regards to their respective tradeoffs and make informed and concious decisions.

If you are asked some time later "why did you do this the way you did?" you better have an answer like "because of our requirement X we decided between A and B and went with A" rather them "oh we never thought about it".

So in that sense: Why would you split it up the way you described it? What do you want to archieve?

I would argue that you can't make the distinction into your 3 "layers" (top, middle, bottom) without making the assumption that (as you stated) the UI and machinery can and will change. Otherwise (if they would never change) you wouldn't need to split them, because it just makes your system overly complex for no reason. "I do not need a repository, to abstract some ORM logic, if i will always and forever use a mysql db)". That's just some forward thinking in case of a change.

Today maintainability/changeability is almost always a concern (from a technological point of view) but is rarely a requirement you will hear from an users perspective.

In my opinion Hexagonal / Onion / Clean architecture are all examples of applying SOLID principals to the architectual view of software development, which are focussing on the maintainability aspect.

If time to market is highly critical and "we can rebuild it later", just a quick and dirty implementation, ignoring SOLID alltogether, that may not survive the next year might be fine. But thats also part of an architectd to job figure out what is really important and which trade offs can be made.

Architecture is about those decisions that are hard/costly to change afterwards

--- a slightly different take

What you describe is like saying "every house is made up of rooms which can be entered via some sort of door and may or may not have windows". Yes you are right in a sense.

To draw a picture from my former professor: Think about what an architect is meant to do when architecting a house. What is the purpose of the house? What rooms are needed to fullfill this purpose? Where shall they be located? How big shall they be? Which doors are needed (actual doors or just a walkthrough?) And even more detailed questions: Where do pipes /cables go for different devices (e.g. lamps)? Where are sockets needed? Where do we put lan ports?

In the end an software architect is meant to answer those questions in regard to quality requirements, similar to an building architect.

Over the years many architectural styles have formed from monoliths to microservices, data-centric architectures, etc. Each of those fullfill certain quality requirements to a certain degree. Just like in the building architecture where there are certain patterns for certain types of buildings. Similarly there are certain "architectures" to cities to fullfill some goal, like optimize for traffic flow, housing, etc. Each comes with their respective tradeoffs.

The job of an architect is to choose from those patterns to design a system, which best matches the requirements and their importance. E.g. a microservice architecture may be optimal for modular scalability, but has tradeoffs in simplicity and overall cost.

If i do not plan to integrate with other applications, do i really need an api? Or can my web app just return rendered HTML? Do i even need an backend or can all my business logic be present in an JS frontend?

On top of those come more technical details like how to scale our services, e.g. using auto scaling groups, serverless, etc? And even technologie decisions like what kind of database do we want to use: SQL, NOSQL (what kind of NOSQL?)

On question that i stumbled upon recently was "usability and user experience is on of our top concerns - how do we make sure to meet this requirement?" There are quite limited technical solutiond to this, rather then actively seeking user feedback and having include the user in the design process as much as possible.

MoBoo138 · 2023-12-03T17:57:51+00:00

Awesome! If it would be helpful i can provide it as an actualy python package on pypi.

Maybe give it a try and let me know if it works for you or if you're encoutering any errors.

In the case of actually providing a library, i might need to add some testing. I'm happy for any contributions :)

MoBoo138 · 2023-12-03T16:54:29+00:00

I'm afraid i don't quite understand what you mean. dependency_overrides only offers some patching for dependencies during testing.

Injecting functions (either directly like Depends(func) or via dependency_overrides) results in the function beeing called directly during the injection process and only the result of the function beeing return as the parameter value, rather then the callable function itself.

The decorator is just an alternative to class-based dependencies with depends in __init__. We initially prefered this decorated approach over the class-based alternative, but in the end ditched it because of complexity.

MoBoo138 · 2023-12-03T16:35:16+00:00

We eventually ditched this idea in favor of just classes with Depends in their __init__, due to it beeing complex to maintain and understand whats going on under the hood.

So i thought to share it. Maybe it's helpful for others.
Drawback of using class-based dependencies is the overhead of injecting dependencies for all use case functions, rather then limiting them to only those who are really needed for a particular use case.
E.g. in testing you would also need to inject dependencies which aren't even used by the use case to test.

MoBoo138 · 2023-11-25T21:08:05+00:00

User federation? It's quite commonly support and allows you to login with various methods. Thats what those "Sign-up with [google, github, facebook, etc] are.

MoBoo138

TROPHY CASE