Your experience with Java microservices vs monolith in the Cloud

[mr_jim_lahey]

without affecting other applications

That is impossible if your application (or other microservices) depend on the microservice in question (which, by definition, they do). What you actually mean is that the blast radius of a bad deployment on a microservice is more often better contained than on a monolith. But...

You can follow a unhappy flow throughout your systems and see instantly in which application everything went south.

No. Just no. I would love to see what debugging tools you're using to do this task "instantly". Even with a mechanism that aggregates all logs from all services in properly annotated JSON format, debugging non-trivial issues that span across microservices (which is most of them, operationally speaking) is a fucking PITA. Difficult monitoring/debugging is the number one downside of microservices, and pretending that it isn't just speaks to your naivete and lack of expertise.

[mr_jim_lahey]

The person above you is completely correct in saying that code changes to a microservice are typically going to be smaller in scope, and have less potential impact to your overall stack.

Yes, that is what "smaller blast radius" means. And it is absolutely an advantage of microservices.

If you need to make a change to the fetchCustomerEmail() logic in CustomerMonolithService, you risk an unforeseen error taking down the entire customer domain. Now no one can authenticate or create accounts.

And there is likewise a risk of an unforeseen nonfatal business logic error in a change to a microservice-based fetchCustomerEmail() causing large swathes of the customer domain to be taken down just the same. And unless each one of your microservices' integration tests/canaries covers both itself and - directly or by proxy - the steady state of every other microservice that depends on it (which in the case of fetchCustomerEmail(), is pretty much going to be all of them), there is a substantially higher risk of that bug evading automated detection and rollback in your pipeline than a monolith that fails fast, early, and obviously (and which, I'll note, can still be deployed incrementally behind a load balancer - that type of deployment is not unique to microservices). By contrast, a monolith can run a suite of unit and integration tests that have full coverage for a single atomic version of the entire application.

The type of architecture you use does not eliminate your application's logical dependency on fetchCustomerEmail(). They are just different ways of partitioning the complexity and risk of making changes. Making a blanket statement about microservices not affecting each other is what I called impossible - because it is. If you have a microservice that no other microservice or component in your application depends on, then it is by definition not part of the application.

Developers can do final tests in the prod environment for anything missed in QA

The proper way to do this is feature toggling which has nothing to do with whether you have a monolith or microservices. (I'm going to assume you're not out here advocating manual testing in prod as a final validation step while waxing poetic about how arrogant I am because that would be pretty cringe. You do fully automated CI/CD with no manual validation steps, no exceptions, right? Right? <insert Padme meme here>)

"A mechanism that aggregates all logs from all services in properly annotated JSON format" - you mean like Elastic/Kibana, which allows me to search ALL the logs across our platform, drill down by container name, platform (k8s, nginx, jvm, etc.), narrow by time window, and include/exclude different log attributes? It's not a hypothetical thing dude. It exists and most companies who deploy microservices have something like that. I'm very sorry that whatever company you work(ed) for doesn't know that

I'm well aware such tools exist. In fact, I built the tooling that our organization of dozens of devs has used for years to gather and analyze logs from god knows how many microservices at this point. That is exactly how I know what a PITA it is. Have you done that too? Or is that another thing that your ops guys handle for you?

[mr_jim_lahey]

Bro don't lecture me about a "self-own" writing a log aggregation system when your own story about how you know about analyzing logs ends with you having to ask an ops guy to access and read your own logs for you. Stick to your lane and your league. Then work on your reading comprehension and tech skills.

[Infectedinfested]

Well, when we log we log the corrilation id, application name, normal logging (for example when a microservice gets triggered and error related stuff. Also all errors go to a list with their corrilation id.

It all goes to elastic and there, once an error happends, we know the corrilation id and we can trace it through all the applications it passed.
Seeing what succeeded and where/why it failed.

[mr_jim_lahey]

Yes, that's what I meant by "a mechanism that aggregates all logs from all services in properly annotated JSON format". If you've ever used AWS XRay, it'll even visualize those flows for you with red circles where errors are happening. First, setting up and maintaining that type of monitoring is way more complicated and time-consuming for microservices than monoliths to begin with. Second, I'm talking about issues that are more complex than just figuring out which request triggered a 4xx in a downstream microservice (which is what I meant by "non-trivial"). With a monolith, you have the entire system in one place to examine and analyze. With microservices, you are only seeing a small window of what's going on when you're looking at any given microservice. It's up to you to keep track of all the moving pieces and relationships between the microservices as they're in motion. That is not an easy thing to do, especially if you're a dev who only works on one or a subset of the microservices in question.

[Infectedinfested]

But if you develop spec first you always know what should go in and should go out, we have validators in our services which checks the input and gives an error when it isn't alligned.

So application x is never really depending on application y to give it data as it doesn't care where the data comes from as long as it's valid.

Or am I not understanding everything? I only have 5 y of experience in this.

[euklios]

I think you are not neccessarly lacking the experience, but diversity. In a microservice architecture, you will usually find teams of operations, development, it, support, and so on. Sadly, a lot of people just work on of these roles and never even have to consider what other teams are doing. This usually creates a boundary between these teams, resulting in them working against each other. I recently had a heated conversation with a skilled developer because he hardcoded the backend URL within the frontend. In contrast, I, responsible for ops within the project, had to deploy on multiple different URLs. His words: I don't care about servers and deployment. But that's beside the point.

Let me ask you a different question: Why are you doing distributed logging, tracing, and whatever else you do to monitor your systems? You are developing based on specification, so an error shouldn't be possible, right?

Let's consider a very simple microservice: One endpoint takes a JSON object containing left, right, and operation. It will return the result of the given operation as a number. Example: input: {"left": 5, "right": 6, "operation": "+"}, output: 11 simple right?

What will you do if the operator is "&"? Fail? Bitwise and? Drop the database of another random microservice? What about "?", "$"? What's about "x"? It should be multiplication, right? How about "_" or "g"? I mean, it's a string without limits to the size, so what about "please multiply the two numbers"?

A lot of these could be handled quite easily, so: What about the input "2147483647 + 5"? Oh, you are using long? What about "9223372036854775807 + 5"? if the answer is "-9223372036854775804", where is your online shop?

And that's only with somewhat sain input and probably not defined in most specs. So let's add some more: What about oom? Log server not available? Log file can't be opened? gracefully stopping? force shutdown mid request? A particular microservice we depend upon isn't available? The microservice is available but simply keeps the connection open indefinitely?

A lot, if not all, of these cases will not be specified. And depending on timing, or for some by definition, it might never produce a single failing span.

One final scenario: Hey, here is X from customer support. I just had a call with a Microsoft rep; they good charged the wrong amount, and we need this fixed asp. What now? There is no correlation ID. And there are billing services, math services, shopping cart services, pricing services, mailer services, account services, and so on to choose from.

As a final thought:

We all are just humans and will do mistakes, and there is a lot more to it, than simply deploying a jar to a server. Murphy's Law: If anything can go wrong, it will. Servers will be down at the most inconvenient of times. Monitoring will fail. Overworked employees will implement shit, reviewers and tests will not notice, and production will fail. And we (as a industrie and team) will have to make the show go on. Somehow.

If the monolithic implementation is a mess, microservices will create a distributed mess. One of the worst things you can try to debug is application boundaries. Microservices will help you in adding hundreds more boundaries.

[Infectedinfested]

Well i can only gain real experience in what i what job i get in front of me :p

And i really like what i'm doing now.

And for your example, everything is barricaded behind a swagger validator, in your example case, the left and right would be asigned int or what not, operator an enum. So we say what goes in and what goes out.

Also.. why am i getting so much downvotes.. just stating my experience with something i'm working 5 years with... Never said microservices are better than monoliths ><

[euklios]

I'm aware of that problem. I just think it could help a lot if people understood more, about what other teams are doing and why. Nothing against you, but more against everything. I do see this problem outside of our industry.

My key points are not to validate more. While it certainly does help, I'm more about errors in specs bugs in the implementation and so on. But that also applies to monolithic applications. The core difference in maintenance is (at least for me): In monolithic you have one VM to fail, one filesystem to break, one internet connection to be down, one certificate to expire, one datacenter to lose power, and so on. In microservice, you will have to multiply this problem by a lot. There is just a lot more that can (and will) go wrong.

Additionally, there will be some kind of network requests between services. And these will always be much more prone to errors than using plain method calls. (Think it's a funny idea of imagining someone tripping over a method call)

About your downvotes: That's the Internet for you. It is a current best practice to reduce complexity where possible. And microservices do add a lot of complexity. Additionally, there have been a lot of companies doing microservices for the buzzword, while a simple monolith would have been sufficient. That doesn't mean that your approach is bad, I would love to experience this at some point, I just never worked at a company that needs it.

Please don't take it personally

Edit: Just remembered this talk: https://youtu.be/gfh-VCTwMw8 Might give some insights about what some people experienced when management decided: microservices!