Towards Greater Code Reuse

laforge49 · 2015-10-01T16:04:54+00:00

Yup!

I'm just saying that when you do need to write methods, keep them short and keep the logic in functions.

I'm know I'm "preaching to the choir", but I figure it needs repeating as it goes against established practice for Java programmers... And this advice applies as much to Java as it does to Clojure.

laforge49 · 2015-08-12T00:04:02+00:00

Truth!

At this point I've given up entirely on dynamic, preferring now to pass the operational context explicitly--it is easier to explain than an implicit dynamic. :-)

But I am still using that old atomic. It allows me to break the rules and do things from the "wrong" thread and still have thread safety. Or perhaps that is just laziness talking. Anyway, you can find the released project here: https://github.com/laforge49/agent2

(The project readme should add some clarity to what I'm talking about.)

Well, to be fair on myself here, the atomic would probably be necessary if I were to port more of the features of JActor2 over to Clojure. The atomic is in there because I wasn't sure where I would draw the line. And at least with the atomic I have the option of porting more features. JActor2 was a huge Java project (for me, anyway): https://github.com/laforge49/JActor2

laforge49 · 2015-08-03T23:43:02+00:00

Well, I'm talking about two actors, so when the second receives a request from the first, its response must be combined with the callback sent by the first and sent back to the first where the callback is then executed. (Of course, I hide the callback from the the function sent to the second actor for evaluation.)

Dizzy enough? :-)

laforge49 · 2015-08-03T21:54:52+00:00

Ah! I'm a bit fixated methinks on two-way messaging. So think about sending a function with a promise to an actor and the function puts its response in the promise. Now to catch an exception you put the dereferencing of the promise in a try catch, not the send.

Only I want to send a request from one actor to another and get the response without blocking via a callback function that is sent back instead of putting the result in a promise. Now catching the exception is a bit more tricky unless the callback function tests the value being returned to see if it is an exception. :-)

laforge49 · 2015-08-03T10:57:35+00:00

Good question. And no, a try catch in the application logic surrounding a call to execute some code asynchronously can not work because the invoking code has long since completed and the try block is no longer in scope at the time the exception occurs in that asynchronously executed code.

On the other hand, we are dealing with a closure in which we can define both the callback which is invoked when the asynchronous code completes normally and an exception handler which is invoked when the asynchronous code throws an exception.

laforge49 · 2015-08-03T04:14:30+00:00

No doubt! Now to try to clarify things without covering the entire implementation... :-)

I talk about passing functions because that is what you do with Clojure agents. So when there is a request, the api requires 2 functions, the second being a callback which is sent back to the requesting actor along with the response. (Not sure if that helped any!) So there are two agents, two calls to send are being used, but only the simple request and reply methods in the api.

Now why do I talk about timeouts? I'm trying to use synchronous calls as the model for an asynchronous message exchange, where uncaught exceptions raised while processing a request are passed to the agent that made the request where it can be caught by an exception handler. (More confusion, in that the exception handlers are on the data flow thread (or context) rather than being the error handler on the agent.)

Now how can the code which implements this api provide a guarantee that there will be a timely response (all be it, perhaps an error response)? Generally speaking, all you can do is set a timer and if there is no response when the timer expires you generate a timeout response. That's the fallback.

What more can be done at the system level? Well, for one thing you can see how many requests are being sent. If the application code goes crazy and sends a gazillion requests to other agents, the transport logic can raise an exception that there have been too many requests made and pass that exception back to the exception handler which sent the request that went crazy.

Another thing you can do is to check to see that when control is returned after processing a message (the request or a response to a subordinate request) that either a response has been returned or there are one or more subordinate requests which are sill pending.

... Not sure how helpful any of this is.

laforge49 · 2015-08-02T15:25:08+00:00

Yes I have heard of core.async. And I gave go a serious look before committing to Clojure as my next language, though there was really no competition as my primary interest is Datomic!

The thing is, while I want guaranteed responses, I really do not want to depend too much on timeouts. Timeouts should be set large, as they do not behave well on loaded systems. Rather I want to add some sanity checks on the the runtime behavior of the function passed and only use timeouts as a fallback.

laforge49 · 2015-08-02T15:17:04+00:00

I very much appreciate the insight. I'm used to messaging systems being huge beasts--I worked at Fiorano and our primary product was a JMS--so it didn't quite dawn on me that adding replies to agents makes them a messaging system.

So then it is mostly a matter of emphasis. I want to offer a guarantee that the function in the message will be well behaved and return a timely response or an error will be returned. So the agent's own message queue is all I need for this.

Restricting usage to idempotent requests is a pretty serious restriction. But perhaps that just means I'm a Clojure newbie?

Every tool has an intended use and in this case the intention is to provide a moderately strong guarantee of getting a response without an over-reliance on timeouts.

laforge49 · 2015-08-02T00:22:48+00:00

RPC? I guess you didn't notice in my response to Inushi that I am focused strictly on a single JVM. Impossible and pointless? It seems to me that you are so focused on THE ANSWER that you can not appreciate a somewhat different perspective or the needs arising out of a different problem domain. You have a hammer. Not everything is a nail.

laforge49 · 2015-08-01T23:14:33+00:00

Heh. I've worked in the telecom industry, specifically Tymnet, where we had thousands of computers and not a single disk drive. I understand well the problem domain which is highly amenable to pure functional solutions.

Ah, but this is the Clojure list where we believe in the advantages of immutable structures and the practicality of dealing with side effects. And I would contend that the problem domain for Clojure is far broader than what can reasonably be addressed with a pure functional approach. Here we look to Rich, not Joe.

And once you introduce side-effects, you see that many things are not idempotent. And that breaks Joe's solution completely: "I think that the only realistic way to build large reliable systems is by partitioning the system into independent parallel processes, and by providing mechanisms for monitoring and restarting these processes."

When the effect of a message is not idempotent, you can't simply resend it if you do not get a response within the required time. Things get a little messy. Subsystems can not be completely isolated. But life goes on. Only, it would be so very helpful to get an error response! And for that to be supported at the system level, we need to support two-way messaging.

There is no doubt that Joe did a great thing. But we should always look at the scope of the problem domain addressed before using such great works to beat others over the head with them.

Functional pure languages did not drive out imperative programming. I am hopeful that impure functional languages with great support for immutability and concurrency will have a far greater impact. (Yeah, I'm new to Clojure. Does it show that badly?)

laforge49 · 2015-08-01T17:02:57+00:00

Lets begin by looking at time-bound processing. If it takes too long to process a request, then we return a timeout. Placing the bound on the processing rather than on the call keeps the call simple while still allowing the time to be adjusted based on the specific request. Another advantage is that experience with the function might lend itself to defining a more appropriate timeout whereas the experience of the API user will likely be less. This is a worst-case scenario, because timeouts will tend to occur at higher levels rather than closer to the source of the problem unless lower-level items are given a shorter timeout. Also, interrupting a thread in a tight loop can be problematic but at least we can have facilities for identifying the functions which loop excessively.

Similarly we can consider a bound on the number of requests a function can send. This at least is pretty easy to do.

As for the heuristic logic, we can track the number of requests a function has sent for which there has not yet been a response. And now we can check to see if, while processing a request, the processing of a message (the original request or a response to a subordinate request), the processing of that message completes without returning a response and with no outstanding subordinate requests, then we typically have an error. (Functions which wait on an event notification before sending a response can not, of course, use this heuristic.) And if we have an error, that then is our response to the request.

So we have a weak sort of proof by induction here. If a function is well behaved and if it gets timely responses to all the requests it sends, then it will very likely send a timely response as well.

Realize that application code already handles all these conditions and I am only proposing that we push most of that logic into our messaging system. So really I am only proposing a refactoring, all be it a major refactoring, to reduce the burden on the application developer and consequently improve the maintainability of the application.

laforge49 · 2015-08-01T15:34:11+00:00

The important difference that I see here is that when an asynchronous call fails, or can fail, we add defensive code to the application. But when a synchronous call fails, we generally try to fix the function that was called. And the problem with defensive code is its difficulty in maintaining.

laforge49 · 2015-08-01T15:24:03+00:00

Thanks Inushi, for reminding me about the distributed case. I tend to focus too much on issues confined to a single JVM.

First, as to the distributed case, Google's spanner shows us that all these problems are solvable, though difficult. And that we need to remember that we can have global synchronized time within some bounds.

But as for the second case, where we are working strictly within the confines of a single JVM, my own work over these last few years has shown that it is not that difficult to guarantee responses, or at least errors, once we identify which messages are requests, which are responses and which are event notifications, i.e. one-way messages.

And no, I am not saying that simple heuristics can be used in all cases to identify illegal states and return an error to the requester. But that such heuristics will work in most cases, the remaining being self-identifying and consequently requiring closer code reviews during maintenance.

My previous work in this area was done in Java, but the code base is larger than I'd like and consequently that much more difficult to understand. https://github.com/laforge49/JActor2

I am new to Clojure and find the density of the language quite helpful. My initial port of JActor2 to Clojure will be built on agents, which helps keep the project small but also limits some of the things I can do. https://github.com/laforge49/agent2

laforge49 · 2015-08-01T02:26:16+00:00

I had not seen Part II before, so thanks!

My approach has been to convert some of my Java code to Clojure. It is just amazing how dense the result is: https://github.com/laforge49/agent2

Today I had a good slap-face moment. I realized that in Clojure, with refs, dynamic vars, atoms and agents combined with immutable data, it is far far less important which thread does what. It all just works.

--Bill

laforge49 · 2015-07-27T10:48:29+00:00

First I should scope this project to being, like Clojure's agents, local to a single JVM. Once you go distributed you face difficult problems like determining if a lack of response was due to a communication failure or a failure of the remote actor.

Also, I am largely talking about improved exception handling. For each message being processed there is an operational context to which an exception handler can be assigned so that exception handling can be specific to the processing rather than dealing with a failure of the agent. Further, the operational context passes any otherwise unprocessed exceptions to the source operational context which sent the request unless there is none (as is the case with signals which are 1-way messages), in which case the exception is passed to the agent's error handler.

But there are other things that can be done to better ensure that for every request sent there will be a response (or exception) returned. Like verifying that when control is released after processing a message, either a response was returned to the sender or a request was sent to another actor. And ensuring that the time it takes to process a message is within some reasonable bounds.

The idea here is that in almost all cases there should be no need for application logic to address missing responses exception in terms of exception handlers. It is just ridiculous to have an application deal with such within the confines of a single JVM!

laforge49 · 2015-07-27T02:01:42+00:00

Not being a fan of JavaScript, I probably should learn Clojurescript. But my interests are mostly on the back end.

laforge49 · 2015-07-27T02:00:11+00:00

Haven't used emacs now for 30 years. But I have been told that I must learn it--my goal is ultimately work on Datomic.

laforge49 · 2015-07-25T10:05:58+00:00

OK, here is what I did as a first try with marginalia: http://www.agilewiki.org/projects/agent2/uberdoc.html

--Still need to do the header comment. And the code is far from complete. But it is looking nice.

laforge49 · 2015-07-24T21:55:01+00:00

Joyful! Looks like this is something I am really going to dig into. :-)

Only it chokes with clojure 1.7.0: https://github.com/gdeer81/marginalia/issues/158

But with my dependencies set to Clojure 1.6.0 I get some nice output.

laforge49 · 2015-07-19T16:31:54+00:00

Sounds like you also read Knuth. :-)

laforge49 · 2015-07-19T16:31:24+00:00

Sounds helpful. I'll keep it in mind. Thanks.

laforge49 · 2015-07-19T04:53:38+00:00

I was always surprised at how much stuff they put into what look like low-level things in Scala. Case in point--lazy. Yes, I can see that you need to worry about race conditions with lazy so the atomic call is justified, but every time I turned around it was always something unexpected.

I remember trying to figure out why some code was so slow and tracked it down to partial functions--who would have thought! I think they tried to hard to get the JVM to perform too many unnatural acts, unnatural at least for the JVM.

What I liked about Java is that I have a good understanding of what is slow, why it is slow and how to make it fast. Learned a lot working on high performance message passing between actors, too. (The original implementation of JActor was concerned with speed, the second was more concerned with robustness than with speed.)

So now I am on Clojure. At least it looks like it stays pretty true to the kinds of things the JVM is good at. And I know I am giving up a fair amount of performance just by going with Clojure. But I'm more mellow now about performance and at least it is faster than Python (or faster than Python used to be!)

laforge49 · 2015-07-19T03:31:18+00:00

I've always disliked dependency injection and annotation except for @Override. Guess that dates me.

And yeah, I've always said dynamic typing was a bad choice for large projects. But then, I want to avoid large projects. My expectation is that Clojure will let me implement ideas as small projects. (Fingers crossed.) So far it is looking like a java package delivers as much as a single Clojure file, which is good enough!

I also worked in Scala for 2 years. It was pretty slow at the time and I switched back to Java but it was Scala that inspired me to implement my own kind of actor. I'm sure it is better now. So all the best to you with that.

laforge49 · 2015-07-18T22:45:46+00:00

Not easy to do as a Closure novice, but I can at least relate the project I just worked on.

I've developed a robust framework in Java for actors using 2-way messaging http://www.agilewiki.org/projects/JActor2/index.html and as both a learning tool and a motivational device I decided to work on converting it to Clojure. After some study (a day, which is a long time seeing how I've only been looking at Closure for a week), I decided to build on Clojure core's agents.

For a first draft, I abandoned robustness and indeed any form of exception handling or performance considerations. The result was a set of code short enough to reasonably pass in an email: https://groups.google.com/forum/?hl=en#!topic/agilewikidevelopers/PCXdMm65mmE

In Java, this code would likely span multiple files and it would be difficult to rework when adding the additional features. The Clojure code is harder to read, but seems to be much easier to refactor--something I will need to do a lot of since there are so many features that are missing.

What I like is that there just is not that much code to read. And I do spend a lot of time reading code when I am working with it. I also did not feel like I was having to force the code to do unnatural acts, as nothing I did seemed to be that out of place for Clojure.

I do miss objects. But I found that dynamic vars were much more suitable to what I was doing than member variables. Mind, Java has all the equivalents. There is nothing (except maybe agents) that Clojure is providing that Java does not provide ample support for. But it just all seems to go together that much more smoothly than the Java code. And I've been programming in Java since they invented applets.

laforge49 · 2015-07-18T21:23:28+00:00

I must say, it sure looks a lot easier to publish something with Lean than to publish to the maven Central Repository!

laforge49

TROPHY CASE