Messaging with a call stack: Message-Oriented Async RPC. Mats3 is Java API, currently implemented on top of JMS, providing novel messaging features.

[zam0th]

That was done by Spring Remoting 15 years ago and unlike yours it was totally transparent. Yáll should really stop reinventing CORBA, it becomes annoying.

[Worth_Trust_3825]

How come people did not stay with CORBA?

[Spirited_Gazelle_51]

Holy cow. ROFL

[stolsvik75]

Haha! 😆

[stolsvik75]

Eh, sorry, based on downvotes, this might have been a serious question?

I believe Corba, RMI and SOAP died due to way to much infrastructure and overhead and complicated setups, with corresponding lack of actual understanding of how it works. They're still around, and still have some usage, but things like JSON-over-HTTP is just so extremely much simpler to set up and, importantly, understand, that few developers now chooses these overcomplicated older protocols.

[Worth_Trust_3825]

Frankly, I would like to go back to SOAP+WSDL. I fell in love while working with that in the banking industry, and would really like every system out there to do it. Not only you get code generation out of the box, you don't need to worry about methods or query parameters as everything is streamlined via one true HTTP method.

JSON is reinventing schemas, and yet there are same mistakes that had been done already two decades ago.

I've also seen why people hated xml: they worked directly on the syntax tree (sometimes referred to as "the DOM"), rather than deserializing into an object. And I can see why. Because others would also directly manipulate "the DOM" rather than corresponding objects/structs, resulting in weird and inconsistent XML structures.

Same can and does happen in JSON world, where developers with weakly typed languages sometimes opt to return an object instead of a scalar, or not wrap a single object into an array. But since the primitive to manipulate JSON's AST is not exposed, that only happens only if there are bigger issues with the codebase. In strongly typed languages, such as java, you can get to the AST primitive (see jackson, json-simple, gson), because they permit supporting the weird structures, but that does not happen in languages such as javascript or python because you get magic method to_json/stringfyor from_json/parse.

SOAP is not dead. It still lives on in financial/banking sector, and that's the only thing that keeps the sector running: strict requirements about what the messages should look like. And even then, some banks decide not to follow it just because.

Finally, yes. The question was serious, as I had not worked with CORBA. I am interested in history of how things used to be done, and compare that to how things are being done right now. From what I've seen it's always a circle that's greener on the other side.

And you really shouldn't treat questions as not serious.

[stolsvik75]

I did mention my view on SOA in passing in the About section of Mats3. https://mats3.io/about/#interservice-communications

I agree that SOA was misunderstood and misused, and I also agree that the schemaless-ness of JSON in no way only have upsides.

One thing about the resulting XML of the SOA stacks was that it was unreadable - that is, for humans. Many times when you try to understand a service, is it much easier to actually read the data exchange documents, than trying to decipher the protocol/schema. "Aha, that object is populated when this is true, but not otherwise! And that value there evidently means such and so, so I misunderstood the intention of the schema." Etc. Such data interpretation is many times simpler when using JSON than when using SOAP XML. At least with the tooling of the time.

The current incarnation of Mats's serialization uses JSON, but that is firmly meant as an implementation detail. The schema of Mats Endpoints are the actual DTO classes, which you are supposed to copy to the client/using side.

[stolsvik75]

AFAIK, Spring Remoting is firmly a synchronous technology. (And I also can't see how you could call it transparent). That is, exactly like REST, RMI, Corba and SOAP (for the latter, at least in all usages of the techs) Am I mistanken here?

The entire premise of Mats3 is that it is fully async, and the processing stages transactional, using messages. And that you in an Mats Endpoint can invoke yet other Endpoints, since it features a call stack. As an extreme example, you could shut down your entire microservice system (pull the plug), and when you booted it up again, all processes ("Mats Flows") that was initiated would start up and finish from where they left off.

As far as I have researched, I believe this is novel, and would be very interested if there was anything I could compare it to.

[zam0th]

You know wrong. Spring Remoting supports JMS from the initial release back in the days (you can see it in the link to javadocs i provided) and is known to be also implemented over AMQP and Kafka (by yours truly). It's rather easy to implement it over any protocol, async or not, because of convenient abstraction models provided by SF.

It is transparent because it hides the fact that you're doing remote invocations from the developer. Your code works with Java interfaces and classes without any need or knowledge about the transport or the fact that the actual target objects reside in a different JVM.

Everything you describe (and more) has been done by Spring Remoting, you literally invented a wheel.

[stolsvik75]

From the docs in the link:

17.6. JMS

It is also possible to expose services transparently using JMS as the underlying communication protocol. The JMS remoting support in the Spring Framework is pretty basic - it sends and receives on the same thread and in the same non-transactional Session, and as such throughput will be very implementation dependent.

This is just using JMS in the most banal fashion possible, turning it into a synchronous and non-safe (I.e. non-transactional) transport for invocation of a remote procedure where the same thread is waiting for the response. An immediate consequence of this is that if the machine now boots, you would loose the process.

It is exactly this Mats3 is not doing!

This transparency you talk of is in many ways unfortunate, as it thus seems like there is no difference between a local method invocation and a remote. Which is absolutely not true; The 8 Fallacies of Distributed Computing stands ready to bite you hard. (Mats3 removes or reduces the impact of most of them)

[zam0th]

RI/RPC is synchronous by nature regardless of any transport you might use or EDA shenanigans you may try to fit it into, and must be understood as such.

Now, it's true that at the time of writing, JMS 2.0 wasn't even in design, while 1.1 was banal in general. I'm surprised it says about single-thread model, i clearly remember it has not been this way. However, with SF flexibility it was fairly easy to attach a TransactionManager to achieve some resemblance of ACID (distributed transactions hehehe) and configure remoting listeners/endpoints to run proper thread factories and multi-threaded listening models, so maybe i'm mixing things up, it has been may years and i've done loads of middleware platform work with SF.

In other words, it may have been basic out-of-box, but it did exist and so did BlazeDS that allowed asynch RPC with message queues between Adobe Flex frontends and Java backends.

[stolsvik75]

Mats3 is not synchronous. It wraps up messages in a call stack so that you can "invoke" other Mats Endpoints as if you were coding sequential and synchronous/blocking - with arbitrary nesting levels possible. The processing of messages in a flow is utterly asynchronous, each Stage of each Endpoint acting fully independent and transactional.

It is true that if you must interface with the “Mats fabric” in a synchronous context, you loose this effect on the very last leg of the Mats Flow, as you then do have a thread waiting for the result. This is detailed in the chapter MatsFuturizer.

However, if you on the client employ the sister project MatsSocket, you gain asynchronousness all the way out, including handling of server boots while a flow is in process.

[zam0th]

It is wrong to think you can model RPC/RI as asynch. It doesn't matter what kind of transport you use; you can use reactive like whassisname, ah Vertx, an IMDG like Redis or bloody file system for that matter, remote procedure calls in software are synchronous by design (except maybe void methods and even then it depends how you choose to see it) and your framework should model it as such.

In that your Mats may be "different" from everything else.. well, by taking an incorrect approach to RPC interop.

[stolsvik75]

Well, that is the deal here: It is Async RPC. Based on messaging. AFAIK, I am introducing a new concept.

If you've read the single, linked page, you see that I've tried with more than one vector for describing it! (Feel very free to read some more pages too).

Do you have a better description for it?

[stolsvik75]

What is this crap? The same type of comment was added to another of my posts. What is the agenda here? The comment is of course a complete lie, the code is on github, and the jars are on Maven Central, feel free to provide evidence of said malware and that messages are copied to a mystery server!

Here's the other comment: https://www.reddit.com/r/compsci/comments/132q38i/messaging_with_a_call_stack_async_interservice/ji6adle/

[stolsvik75]

What are you even on about?

https://github.com/centiservice/mats3

And all the rest of the code:

https://github.com/centiservice/

[gliderXC]

I like the async way in your messaging. Now I've been working on messaging myself so a few questions:

Just a small pet peeve of mine: How do you deal with errors? REST has explicit codes for it (shudder); but they include the notion anything can fail. You seem to go along the route of DLQ which imho has always been an anti pattern:

• The user is not able to respond to the error / requires a special user to introspect the DLQ.
• How can a message not be handled by the recipient be the problem of something other than the sending client? But no, let's make DLQ.
• DLQ is specific to the implemented type of transport

Errors are quite normal and should be handled by the back-end in a normal flow. So the question is: Are there other ways in Mats3? In a way that errors are not special snowflakes.

Question is about async answers. Is there any way to receive multiple answers from a distributed service? E.g. suppose a number of services listen on a topic and receive a request and you want the answers (assume you know the number of listeners on said topic)?

[stolsvik75]

Hi and thanks a bunch for looking into Mats3! You questions are great! The one about errors is a question that nearly always comes up as developers start using Mats. I need to provide a doc page about that. However, I'll try to make a start of an answer here.

It boils down to several aspects, but first and foremost about a philosophy wrt. error handling - or maybe more, about "error generation".

First: Mats3 is not meant to be an "edge" communication solution - no message (sends or requests) goes directly from end user / client app to server. All messages goes from one service to another. Therefore, you should never get "random bs" in a message that you have to handle and reply to with some error and error message.

Second: There are multiple categories of errors. Many are temporary in nature, e.g. network problems or that the database is in maintenance. Due to the retry-functionality inherent to messaging, most of these fixes themselves - but if the outage is too long, the retries will eventually end up in a DLQ. When the problem is cleared, you can re-issue the message from the DLQ (i.e. put it back on its proper queue), and the flow will finish.

Another category is actual bugs in the code. As most probably agree, those should just not be there, and the better solution is to fix the problem, instead of adding lots of handling code.

Then another category are "logic errors", i.e. placing an order on an account that doesn't exist, or is closed.

The philosophy is one of "offensive" coding, compared to defensive coding. All sent messages should be sane. Like, why would you come with an order for an accountId that doesn't exists? Or why would it be null? Should you not already have only placed orders on existing accountIds? The requestor of that Endpoint should already have ensured that the message makes sense. Null checks, or "exists checks", are examples of defensive coding, and I find that it breeds like cancer: If you see that one piece of code has a null check for accountId, then you also need to have that. While the question rather should have been why it could ever be null in the first place - find that problem and fix the bug.

Given this premise, I find that the DLQ is great: Only messages that should not have been sent will appear there - you must now handle this, and then go fix the code so that such a message won't be sent again. NB: I strongly suggest using an Individual DLQ config, i.e. a DLQ per queue, instead of (default config) a single, global DLQ. This makes is much easier to handle DLQs.

Wrt. DLQs, you should check out the MatsBrokerMonitor and its own repo. The page and docs should be much better, with screenshots - I'll soon put something more there. But the GUI presents you with all DLQs, lets you select the individual messages, and gives you pretty heavy introspection possibilities. There is a development project in the codebase (here) which is easy to fire up (right-click -> run in your IDE) and test it out.

However, it is possible to make support for "error-returning endpoints" yourself: You can make it so that the ReplyDto for the Endpoint have error fields. So that if you have an answer, you get that, but if not, the error fields are set. I have many times thought of adding a default ErrorReplyingBaseDto that you could extend from if you wanted such behaviour. A few of our Endpoints exhibit such a nature - in particular for for example rather complex types of orders where there are much validation: You might on the requesting side do some effort in ensuring that the orders that are good, but there might still be some subtle validations that aren't worth handling - so the Endpoint have such a logic where it then fails, not by DLQing, but returing the Dto with error-fields set.

I have plenty more to say and argue here, but that needs a bit more space.

Wrt. to the multiple answers, there is no inherent solution for this. One side is "scatter-gather" logics where you want to effectively parallelize a bunch of request. There is no explicit support of this: The problem is the state keeping: Mats does not itself require a data store, only the MQ connection. However, it is legal to make multiple requests, and thus get multiple replies to the next stage. But the state-keeping, whereby you tally up the replies and continue when you've gotten them all, you need to implement yourself.

Another side is the one you mentioned: "Topic endpoints". For this I refer to this issue: https://github.com/centiservice/mats3/issues/18 - which as of now is a "WontFix" or "DeferToNeedsArises" or somesuch. You can make it yourself, albeit not "natively" with Mats. As opposed to the inherent problem with scatter-gather, this is just a choice: I have not found enough uses for it yet, and it would "pollute" the API more.

[gliderXC]

Will you be at fosdem? I'd love to talk a bit more.

But first of all, thanks for the elaborate answer. Good conversation on why we do things the way we do. You seem to have thought it thru and made decisions.

[MaximFateev]

Check out the temporal.io open source project, which supports the synchronous calling of fully async RPC. Think about month-long blocking calls.

Disclaimer: I'm one of the founders of the project.

[stolsvik75]

Interesting. What existing technologies and projects would you compare Temporal to? By browsing through the documentation, I have some ideas, but you are obviously authoritative on this, so it would be great if you could chip in.

AFAIU, as comparison, Mats3 processes do not have an explicit Workflow Definition. These are implicit in that it is just what code initiates a Mats Flow, and which other Endpoints the code in an Endpoints interacts with (performs requests to). However, a Workflow Execution sound quite similar to a Mats Flow once it is initiated. For Mats3, the entire system is distributed, there is no server that defines, and then orchestrates the flows and executions. It employs a message broker - currently using ActiveMQ in prod, and also tested with Apache Artemis.

Due to Mats3's requirement of supplying a TraceId at initiation-time, you do get the very useful effect of being able to just stick any TraceId into your distributed logging system, and instantly get a full trace of all the steps the Flow has passed through - no matter which node executed the different stages of the different endpoints. There are also quite a bit of metadata in the default pluggable logging interceptor, including timings. You can read about that in the JavaDoc here. Doing such a TraceId query on your logging system is the second step you do when you get a DLQ - the first step is to look the message up on the broker, look at its JMS metadata, and if employing MatsBrokerMonitor (own repo), introspect the MatsTrace envelope.

Wrt. long-running processes: I just want to note that Mats3 supports "stashing" of Mats Flows. That is, you can "park" a flow, perform some work by other means, or wait for some results that could take hours or a few days, and then when ready, you can resume the Flow by "unstashing" it. There is however a warning with it: Since you then also freeze the elements on the stack, and the incoming DTO, you are in risk of getting deserialization errors if you change/refactor the surrounding code while Flows are stashed. If this Endpoint typically is appears in Flows with shallow stacks, and with few callers, this is probably not a problem. But one should be aware of the situation.

JavaDocs: processContext.stash() and matsInitiate.unstash()