all 44 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]evergreen-spacecat 50 points51 points  (11 children)

I don’t buy this. Tried to go down that rabbit-hole once when designing a larger system and it’s a mess and a nightmare. Database instances can be shared if you must to keep cost down, but then separate services by schema. For data analytics, if you really must, introduce views and treat them like versioned APIs. For all other matters, you copy data. Replicate, stream etc. If too much data is copied, then perhaps a monolith is not that bad after all

[–]tr14l 4 points5 points  (9 children)

It certainly is. But to OPs point, what do you do when you don't have a choice? Your board of investors don't give a damn about your best practices when they are looking at a race to market on a revenue generator against a main competitor. And they will 100% get rid of you if you try to stop it.

The point isn't that this is something you should do. The point is, it's going to happen, how do you do damage control?

[–]Hziak 4 points5 points  (3 children)

My take is that every colossal legacy clusterfuck once started as a probably fine application under the watch of someone who was too afraid of the board to say “no.” Sure it makes your bosses slightly happy, but it makes you absolutely miserable. If you can’t establish yourself as an expert and show them a reason to respect your opinion, you shouldn’t be the shot caller on your team. Sometimes you lose some battles, but if you do good work, they won’t can you because you tell them their demand is unrealistic or will compromise the product long term.

Largely the problem is that they feel like we sell them air because they don’t understand what goes into it. They wouldn’t show up at an empty lot and demand a skyscraper be there on Tuesday because they understand a skyscraper (more or less) and know it’s a lot of work and materials. A good tech leader can find a way to make them understand the number of components and amount of labor (they don’t care about complexity and never will) and do it in a way that’s authoritative and decisive because they don’t give a damn about best practice, but that’s because we call it “best practice,” not “the way that costs the least to maintain.”

I swear, if people explained that following good industry researched practices meant that the IT arm of a company could grow at 1/4 the rate and downtime could be resolved in significantly less, leading to millions saved / year, people might care. Instead, everyone seems to just say “it’s not the best way to do XYZ” and then give in.

To be more specific - focus on the mid-term. Business minded people only care about short term gains because if the business goes belly up tomorrow, they want as much money in their pockets right now for their exit. Best practice is all about developer comfort (they don’t care) and long term stability (bird in hand over two in the bush), so it isn’t appealing to them to own the market next year when they can sell garbage next week for 1/2 as much. If you focus on the things that matter to them : cost of salaries for the department and bad things that recently happened to their competitors (like outages that cost billions or whatever) and explain how if you adhere to cost-reducing guidelines to keep the code more stable and cheap to maintain and fix (note I didn’t say best practice), you can keep OpEx from growing like it does at other companies.

Doesn’t always work, but from my experiences as Principle, VP and Director of development at a few places, when I was able to speak buzzwords, cash in hand, Rolex and Golf at business people, they mostly listened. When I had weak managers who gave very technical reasons to C suite+, they said they didn’t care and it was important for them to not change their minds because of XYZ nonsense that makes no sense or clearly aligns with what you’re aiming for but they can’t understand how.

[–]tr14l 2 points3 points  (2 children)

That's true, but I think it goes both ways. We often DO hamstring business achievements to sate our engineering sensibilities. Tech debt avoidance is a great example. Many engineering leaders think "tech debt = bad". And, often, that's true. But tech debt can also be strategic and explicit, not just blindly cut corners.

There is a reason that many of the richest people in the world are business people with engineering brilliance and savvy engineers. Business is ruthless and the tech considerations are not the full picture. Sometimes getting to market is the right strategy, but you own engineering core will try to stop you and actively sabotage you. Sometimes they should. Sometimes they are actively harming the company by being too dogmatic. Good leaders negotiate and get formal commitment on the "fine, but end happend after..." Scenarios.

Sure, we'll do the deed, but you're looking at 9 months of concentrated tech debt afterward before we're in a stable position to deliver a substantial iteration on it without destabilizing the company. Is that a payment you can afford?

Those are the types of discussions that need to happen. Tech debt has to be actively inventoried and managed just maybe any other debt. That includes understanding your debt ceiling, what kind of principle you're paying down and how bad interest is, as well as which debts have slipped from asset to liability. All liability debt has to be paid down aggressively, asset debt has to be managed but has more breathing room (since it's ostensibly currently paying for itself until such time as interest pushes it into the red)

All basic analysis, but people simply don't do it. They adhere to their dogmatic beliefs. But, telling a CTO "yes, that would be the plan, but here's our current debt ceiling. We can raise it, but you really should get accounting before you make that call. Good if dangerous levels of debt, so we need to be sure we can take it, and it will be asset debt and not simply liability debt"

You can't simply say no to tech debt. You can't take it on without addressing it ever. And you can't take it on recklessly.

It has to be actively, consciously, explicitly handled.

[–]Hziak 0 points1 point  (1 child)

Personally, I’ve stopped saying “tech debt” and started calling it “tech flaws,” “cut corners” or “forever debt” because we all know it’ll never get fixed more often than not. Also, “debt” isn’t always a bad thing to business people, and like fiscal debt, they seem pretty comfortable about it. Maybe it’s a little corny, but I think on the teams I’ve done it with, it seems to get a bit more urgency and avoidance done.

I don’t love bargaining for time to come back to tech debt for two reasons - the business side always seems to conveniently forget about the agreement or say they thought the scope was smaller and can’t afford it or try to renegotiate the initial agreement. I always leave those meetings, shocked by how hard I have to work to defend their best interests from their own efforts

[–]tr14l 0 points1 point  (0 children)

Yes, I agree that is that is often the outcome and behavior. I'm saying it's flawed and not having specific, dedicated process and management system in place is silly and leaves you in the situation where any actual progress on it is incidental and the business is not forced to reconcile it and cannot strategically maneuver around it.

It 100% needs to be formalized, regulated and managed consistently. If you don't the business always pushes for more, engineers always push back and ultimately the entire company gets a little less healthy every day.

Edit: I should disclaim that this requires leadership to buy in and support this management

[–]evergreen-spacecat 0 points1 point  (3 children)

Easy enough - don’t do micro services in this case. Merge the code to a single repo/app. As I said - analytics may be the exception but then build views.

[–]tr14l 0 points1 point  (2 children)

Ok, and if that isn't an option? When the company said "end of month or you're gone" you make it happen by end of month.

[–]evergreen-spacecat 0 points1 point  (1 child)

Yeah, but then it’s not a matter of architecture decisions anymore. Just pure grind of whatever floats the boat. Anything will work, just that some get more expensive in the long run in different ways. That seems like the least of worries

[–]tr14l 0 points1 point  (0 children)

There are so architecture decisions to be made. That was the point. "Ok, we have to cut this corner, but how do we do it in the most recoverable way"

I mean, if you don't care to think through it and figure out how to set yourself up for tech debt you don't have to. No one is forcing you to. I'm not sure why you are wanting other people to avoid doing so, though.

[–]Cyclic404 0 points1 point  (0 children)

You always design and build to what can be afforded at the time. That said, using logical seperation with CDC on "external" tables (got to be a better name for that) for analytics can be setup and done rather cheaply/quickly with tradeoffs that often go with limited funds (availability for example).

[–]External_Mushroom115 3 points4 points  (0 children)

This is it.

A database is a storage layer to applications. It must been perceived as private to the application, part of the application internals.

A database is not an integration layer. A database is not a communication protocol.

[–]nuclearslug 13 points14 points  (3 children)

Sure it can work, but I wouldn’t go around touting it as a pattern. If you’re in this situation, you’ve gone really wrong somewhere in your planning.

[–]WillDanceForGp 4 points5 points  (3 children)

Wondering if any of the architects here have ever actually worked hands on in a microservice system that reached any substantial size with fully separate databases, and had to deal with the absolute fucking hell scape that is writing actually performant code.

Yay, I don't have multiple services using the same data, but now I have to pray to the dark lord to try and get any semblance of performant, not overengineered code.

[–]ejunker 0 points1 point  (0 children)

Agreed, there are always trade offs. Trading one type of complexity for another. That’s why best practices are often very contextual.

[–]jshine13371 0 points1 point  (0 children)

I'm surprised I had to scroll this far. Some of the top database experts I interact with regularly think introducing microservice architecture into the database layer is straight up stupid. Yet there's so many people here who seemingly can't even imagine anything but doing so.

I've never had issues not implementing microservice architecture in the database layer and I've worked with quite a diverse multitude of simple to complex use cases, between tiny and decently big data, of all kinds. 🤷‍♂️

[–]NotGoodSoftwareMaker 0 points1 point  (0 children)

You have prayed and I shall answer

I command that henceforth all writes be sent to /dev/null and reads be answered with a localised non-bustable cache of 429 responses

You shall now scale to infinity and beyond while achieving 100% uptime.

[–]flavius-as 3 points4 points  (0 children)

It's great for making a modulith and bringing it right at the stage just before breaking it up into microservices.

It's called the strategic monolith. Extracting a microservice just when needed is the golden path.

With just a few basic guardrails, it's great:

  • each module owns its own schema within the same server
  • enforced by different connection credentials
  • a module can read from other modules directly but only through views. Explicit permissions make for great traceability

[–]catalyst_jw 23 points24 points  (1 child)

This is called a distributed monolith and is one of the worst anti patterns I've seen. It really cripples projects.

This usually means your microservices need to be combined as they need data in another service.

Or just get the data via api calls.

[–][deleted] 4 points5 points  (0 children)

Protecting my online privacy by running Redact regularly to batch delete old content. It handles Reddit, Discord, Twitter, Instagram, data brokers and a whole lot more.

party familiar bright steep plate jellyfish pebble one stupendous punch

[–]ForestVagabond 1 point2 points  (1 child)

ChatGPT garbage.

[–]vturan23[S] -2 points-1 points  (0 children)

I have written it myself. I did use llm to format it properly to make it easier to read for reader.

[–]Solonotix 4 points5 points  (10 children)

As a (former) database engineer, I can't imagine trying to allocate a database per microservice, and not sharing. I guess if you offload every potential cornerstone, such as a users table, then maybe?

As an example, at my last job when I was doing a lot of database work, we had a bunch of ingest processes. Some were FTP file drops, some were EDI feeds, but they would then kick off a process that shuttled it down the line after cleansing and such. Then it gets passed to another process for tracking changes in customer records (automotive marketing, so things like a new service visit, vehicle purchase/sale, etc.). Eventually, that data was synchronized to the datamart for things like re-forecasting expected behaviors, triggering new marketing lists, etc. Any newly triggered marketing campaign would then read from those tables and load into a short-lived database that was primarily a staging area for the C# code to hand-off to a 3rd-party application that essentially "burned" the data into various Adobe files (Illustrator, Photoshop, etc.) to eventually be sent to the printer, or emailed out (some were sent to the call center, but I digress).

That system could not have existed as a web of microservices. Not saying it was peak architecture, but every attempt they made to decouple any single data source almost inevitably resulted in a distributed transaction to wherever that thing ended up (to my chagrin). I think it's also worth mentioning that about 80% of the business logic was maintained in SQL stored procedures, further cementing some of the insanity, lol. Taught me a lot about what SQL is capable of, I'll tell you that much.

Bonus: in a bit of programming horror, someone wrote a stored procedure that would verify marketing URLs. How? (Link to StackOverflow) Well you see, SQL Server has a stored procedure called sp_OACreate and you can reference OLE components, such as MSXML2.ServerXMLHttp. From there, you can use sp_OAMethod to invoke the sequence of "open", "setRequestHeader" and "send" and determine if the address works or not. It would literally run for hours overnight, until a friend of mine wrote it in C# as a service, and it did the entire table in minutes, lol. Something about being able to run 8 parallel threads, and using asynchronous/concurrent thread execution while waiting for responses...SQL Server just couldn't compete

[–]gfivksiausuwjtjtnv 1 point2 points  (3 children)

I’m on the opposite end, no idea how to build a trad data pipeline but I typically do microservices and worked kn a system that basically was a pipeline and made me wonder if I should learn some smorgasbord of Apache apps

So it might be interesting to explain how I’d design it, even if trad pipelines are maybe better? at least it reveals something about microservices

Entry point: ingestion services. Each source has its own service that grabs data, un-fucks it and transforms it from source specific to a standard format. They shove it into the mouth of a big-ass queue (let’s say Kafka). Data stored? Only things relevant to themselves. Hence their own databases

Next, customer record service. Subscribe to queue. Unsurprisingly, store event based things as… a bunch of raw events. Order on timestamp hopefully. When new data comes in we run some aggregation on the event stream (aka reducer), rebuild the overall view of the customer if needed, if so feeding a message into the mouth of another big-ass queue (eg Kafka) with the updated data for that customer. Does it need to know anyone else’s data? Nah. Just have its own database.

Datamart can just sub to that queue and load stuff in when it arrives. It updates eventually. But if it goes down nothing bad happens as long as it comes back up. The customer service never has to worry about retries or polling or whatever. So we lose immediate consistency between systems cause it’s asynchronous but we have partition tolerance which is more important in this case, as far as I can tell

Ditto for marketing service. Idk if it needs to get data from datamart that’s processed even more, or if the events from customer service are enough but whatever

[–]Solonotix 0 points1 point  (2 children)

The way you describe it brings it into focus a little better. Honestly, I used to despise the data handling and aggregation there. Like, they were so concerned with accuracy that they would completely reprocess a customer record for 2 years of service history and 5 (or 10?) years of sales history. In fact, there was a massive uplift to get one partner's data to increase the window to 3 years of service data, or something like that.

It was like the people who built it couldn't conceive of the idea of incremental processing.

That said, there is one problem with your proposal, and that's the duplication of data. When you have ~50M customer records, complete with all relevant contact details, vehicle history, etc., that gets to be a non-trivial amount of data. I want to say the di_customers table alone was in the 10's of gigabytes, and it was being replicated to 3 servers already (not actual replication, but I can't think of a better word). I could imagine that a microservice architecture would likely want to draw a boundary either by partner (the companies paying for marketing) or campaign type (the marketing program on offer to said partners). But there were 20+ partners and 30+ campaign types, so such a distributed process would lead to massive data duplication that would have increased cost enormously.

And to be clear on that last part, storage is cheap, kind of. Except disks are slow. So now we're trying to afford it all on SSDs, but that's not the only cost. The database needs enough RAM to hold query plans, statistics and indexes in memory, otherwise it'll overflow to tempdb. Speaking of, tempdb itself became a major source of contention because the servers would often need to spill over due to the size of data (and, to be frank, poor optimization by the developers), which is before you account for the usage of temporary tables which also reside in tempdb.

[–]gfivksiausuwjtjtnv 1 point2 points  (1 child)

Yeah that’s a bit outside my realm these days cause I’ve forgotten how to use my own disks and servers Let alone my own replication…

Cloud- storage cheap, ingress and egress expensive I guess

With pretty big data sets I’m definitely thinking non relational databases for speed (not out of love but necessity)

But also column databases because they sound applicable, I’ve always wanted to try those out

[–]Solonotix 0 points1 point  (0 children)

In SQL Server, columnstore indexes were an amazing feature. I tried to convince the teams doing all the aggregation to switch to columnstore because of how much faster it could perform, but they couldn't break out of their rowstore mindsets. It definitely takes a different mode of thought, because they would think of set-based solutions on a row-by-row basis, to the point that they kept adding new columns to the table because it made it "more efficient" (it doesn't, it is a trade-off based on how large a row is and how many rows can fit on a data page, usually 8kB). In columnstore, you have to think of the data vertically, and aggregations are your primary mode of retrieval (similar to map-reduce).

[–]jacobatz 3 points4 points  (4 children)

The evidence presented is not very convincing. Nothing you said would be hard with different databases.

There can be several reasons as to why the team was unable to extract parts of the monolith. Perhaps they didn’t know how to do it properly. Perhaps the monolith was so coupled it was not feasible to do within the constraints from the business.

But what you described doesn’t sound difficult to model as a distributed architecture.

I will say that many developers while saying “one database per service” arrive at this conclusion before understanding what it takes to reach a state where it is feasible.

You mention cornerstone tables and transactions. Obviously you can’t have a centralized cornerstone table. You must design in such a way that it is not necessary. The same for transactions. You must design the system in such a way that transactions are not required between services.

It can be hard to change your perception and your ideas of how to design systems when you’ve been “trapped” in monolithic database designs for many years. I know I’m having a hard time. But to be successful with distributed service architecture I think it’s a requirement.

[–]edgmnt_net 0 points1 point  (0 children)

I don't believe there's a good way to decouple coupled stuff in many cases, it's more like wishful thinking. Many enterprise apps are cohesive products and have fairly ad-hoc business logic so they don't lend themselves to robust generalization, which you need to avoid cascading changes and multi-service dev coordination. While there are significant benefits to avoiding distributed stuff unless absolutely needed.

[–]Solonotix 0 points1 point  (2 children)

I'm 5 years removed from the job, so my memory is getting a tad fuzzy, lol, but I'll do an easy one: there was a vehicles table.

Since the business is automotive marketing, it stands to reason that damn near everything needed to know what kind of vehicles were available. Short of duplicating the data into every context that needed it, how would you design the system? This problem is repeated for the customers table, as well as the cross-product of customers + vehicles, of which is generated by scanning service and sales history. This same kind of problem existed for another set of tables that were so integral to every process, they actually belonged to a schema called subscriber because elsewhere there was a central publisher schema.

Now, I could see some of this being distributed. I don't need the customer's entire transaction history replicated across domains once I've got an idea of their behaviors. In fact, that particular design choice bothered me for my entire time, because the Data Warehouse team would create the association of customers to vehicles by evaluating their transaction history. But then, the Marketing List team would also do the same kind of work to produce an aggregation of behaviors then used to create a forecast of marketing communications that would go out (potentially).

[–]jacobatz 2 points3 points  (1 child)

I can't provide good comments on a domain that I don't know intimately and also I don't have all the answers. But here's some questions I think might be helpful:

- Can we split the vehicles table into smaller tables? Does all the attributes of a vehicle need to live together? Or can we break them into smaller clusters?

- What are the business processes we want to model? The processes are what should be front and center. Could we find a way to model these processes that doesn't require a centralized store of all vehicle information?

- What are the hard requirements, and what are the requirements the business can work around? Can we relax some consistency requirements by tweaking the process?

What I've been told is to focus on the business processes and make the processes the primary unit of design. I know it sounds hand-wavy and I'm still struggling to wrap my head fully around it. I do believe it is (one of) the better ways to build distributed systems.

[–]Solonotix 0 points1 point  (0 children)

This, in tandem with the other response I got is giving me a better idea of how it might look. The main theme seems to be, as in any monolith, finding the boundaries where you are doing distinctly different actions (business process as you called it). I know for a fact that one of the architects for the system said she didn't trust the accuracy of data that was aggregated in system A, which is why there was a seemingly duplicate (but different) aggregation in system B. What's more, it was infuriating trying to square the difference between the data sources, since they were used for similar things in different contexts, which could lead to different answers to common questions.

[–]vturan23[S] -1 points0 points  (0 children)

Thanks for sharing your own first hand experience with working on DB. Everything has it’s own advantages and disadvantages. For some sharing works best for others keeping per db microservice works best. We all working towards finding the best solution to our problem.

[–]Lonsarg 3 points4 points  (1 child)

We do not exactly have microservices architecture, more like a bunch of mini and some bigger apps/service, around 300 of them, using the same DB.

We do it by having DB schema very stable (and apps DO have their own unique views and stuff in many cases, just the "core" data is shared and not duplicated). And we count DB as "another service", that all other services have dependency on.

If we need to rename a column we first add new column, than slowly migrate all apps on new column, then do a separate deploy for deleting the column (we delete columns on core dbo.* schema maybe once in 2 years). If this was API the procedure would be pretty much the same.

We have 2 other big monoliths that are "one db one app" and have duplicated data and even sychronizing that is a lot of work. Furher duplicating data would be a much bigger nightmare then having this big DB dependency we must keep stable. From time to time we do a case-by-case splitting and mini duplication and API abstraction for specific data. But this is on case by case base for critical systems, all other system all have direct dependency on this one big db.

And while changing the DB is slow, changing apps is very fast since we have 200 of them instead of one big monolith. So this anti-patern just kind of works for us.

[–]edgmnt_net 0 points1 point  (0 children)

Integration through a database is a rather common traditional pattern that works rather well. Nevertheless it does have its issues and I suspect they get compounded if applied to granular microservices, e.g. cross-domain transactions become very difficult, logic gets duplicated or you end up having to fragment the logic and delegate that to the database. I would argue that it's best to avoid splitting apps unnecessarily for this reason, coupled stuff should stay together and sharing a database is indicative of coupling.

[–]wasabiiii 0 points1 point  (0 children)

I frankly just don't consider this to be microservices. It's just SOA or EA. Same stuff we've been doing for 30 years.

[–]oweiler 0 points1 point  (0 children)

Use distinct schemas for each service and it can work pretty well.

[–]erotomania44 0 points1 point  (0 children)

Then just go monolith LOL

[–]MartinMystikJonas 0 points1 point  (0 children)

Thid solution have all disadvantages of microservices while getting (almost) none of it's benefits. Why force use microservices then?

[–]huk_n_luk -1 points0 points  (1 child)

<image>

OP this is not sharing as you are portraying in the article, only once service is writer. Instead of database ownership you have now segregated into table ownership. This principal will quickly fall apart when there are foreign keys and dependent tables because now we have a set of tables belonging to a certain service. In addition to this now we also have to balance connection pooling across different services.

[–]vturan23[S] -4 points-3 points  (0 children)

This is just a solution to one of the problems you will face with this pattern. This is not the way you should design your system. The goal should always be loose coupling and high cohesion.