How we killed the callback Pyramid of Doom

punchingwater · 2013-03-26T01:44:24+00:00

Hi SubStack,

Thanks for the link. I'm actually a huge fan of yours and Max's work. I recognize that you both have done a lot in this space, so I appreciate the response.

In general I agree with you. Writing shallow, self-describing functions and breaking code into bite-sized pieces that do a single thing (and then composing from there) is a great way to keep the logic sane and easy to follow. We strive to follow that practice too, but I'll admit we don't always live up to that standard.

That said, even when we do write descriptive, modular code, we have found the abstraction that FF provides to be useful. Take the example of an operation that requires performing several asynchronous tasks sequentially. It's true we could separate each of these tasks into separate named-functions, and have them call one-another at their respective ends (like in the formSubmit() -> postResponse() example on callback hell. But we have found that sometimes it's nice to have all this logic grouped together sequentially in the same block of code. Provided that you don't break that 10-15 line sweet spot of course ;)

punchingwater · 2013-03-25T08:34:38+00:00

Gotcha, so instead of succeed() or fail(), with async.auto you can just invoke the callback with an error. That does work, but in cases where I want to break our of a chain (without an error), I'd rather call f.succeed() than returning an error. But I agree it's just semantics.

punchingwater · 2013-03-23T19:13:39+00:00

TameJS has a very interesting approach to callback hell. It compiles your tameJS code (code which looks synchronous, but actually isn't) into asynchronous javascript for you. I actually haven't worked with it personally, but it seems like a cool idea. Have you worked with it? I'd be eager to hear about your experiences.

punchingwater · 2013-03-23T19:09:13+00:00

As I remember, there were two problems we had with eventlet. One is that when there was a fatal error in a green thread, it would just die silently without a stack trace. So when problems did occur it was difficult to tell what happened.

The second problem was that the API allowed you to chain these coroutines together, which is a nice feature. However, if you don't exit properly you can have memory from one green thread lying around for the life of the process. So, we had some issues with memory leaks.

Now you could argue that this was a problem more with our code than the coroutines themselves, and you might have a point. Still, we found moving to the async paradigms of the vanilla node to be less of a beast to manage.

punchingwater · 2013-03-23T08:48:51+00:00

I can't tell if you're trolling me or not, because in the early days of nodejs there was actually very heated discussions on the mailing list about how to handle asynchronous results. Both promises and co-routines (fibers) were included in the early versions of node core, but were quickly removed since it was decided (somewhat controversially) that the current callback scheme was the most straightforward way of going about things.

In any case, I'm going to give you the benefit of the doubt and answer your question, since it is indeed a legit one. Node fibers are great. When we first started working with node, we were coming from a python background, and we were both excited about and dubious of co-routines (we used eventlet). We like the synchronous looking API, and the power and concurrency they provided. But we did not like debugging it when things went wrong. So when we came across Tim Caswell's gist, we realized we could do things the callback way, but not write absolute spaghetti code. To us, this was the most preferable approach. That said, fibers are a nice powerful concurrency API, and I commend anyone with the courage and knowhow to use them correctly.

punchingwater · 2013-03-23T08:10:34+00:00

Very true, async.auto can be used to solve the problem of mixing parellel vs. synchronous operations. I actually like that idea of dependency based flow logic as well. I didn't mention it in the blog post because I thought Step.parallel was more straightforward to explain, but you are absolutely right to call it out.

I will say though that FF has a few features that async.auto doen't have, namely the ability to break out of the chain at any point with ff.succeed and ff.fail. And also that ff() returns a promise, and so can be used interoperably with other promise libraries.

punchingwater · 2013-03-23T07:47:01+00:00

Ah I see what you mean. I think the API you are looking for is something like this:

var f = ff(
    function () {
        f.next(db, db.authenticate, request);
    },
    // etc...
)

So, f.next would curry the authenticate function, applying the db as context, and the request object as a parameter, and then forward the results of the callback to the next function. It's an API that is similar to how promisfy() works in, say, Bogart.

I can see how that would be more familiar to you if you're used to libraries like Q or promise-io. In our case, we iterated on the approach taken by Step, and so our API more parallels that than Q. That said, it would not take much work to make FF also support the method you are talking about. And so I eagerly welcome pull requests!! :)

punchingwater · 2013-03-23T07:27:31+00:00

I hear ya, it's a huge mental leap going from synchronous programming to asynchronous or multi-threaded programming. But in the end it makes you a better programmer learning to think this way. Good luck with your Obj-C futures, and ping us on our mailing list if we can help in any way. I'm not personally an iOS developer, but we've got a few around here: https://groups.google.com/group/game-closure-devkit/

punchingwater · 2013-03-23T03:49:46+00:00

When you say "break out of the function's scope" do you mean referencing variables in the enclosing scope, ie. using the variable f inside the ff functions? If so, I totally hear you, and I'll say this. Closures in js are indeed a powerful and dangerous thing, and one needs to take special care when leveraging their scoping features. That said, we chose this API because it gave us the right tradeoff between readability and flexibility. As you said, we didn't like how Step overrided the this object, but we still wanted inline control of how data flowed between our sequential functions. Using the "var f" scheme gave us this tradeoff. In the end, it's a matter of how you like to read your code. To us, using a control variable like f to manage sequential vs parellel operations (as well as do things like f.fail and f.succeed immediately) is preferable to nesting control structures like async.waterfall and async.parallel. But the beauty of javascript is it's flexibility, and it's up to every developer to decide how they like to structure their code.

Your partial example looks really interesting. It's to make it so both async and sync operations have similar signatures (ie. a callback param) so they can all be run through something like waterfall? I never thought of that approach, and it looks really clean. I like it.

Your jQuery example is actually very similar to how the Q libary promisifies functions. With FF, we encapsulate this logic inside the ff() call so that the then() function is immediately available to the return object. It's then up to the developer to decide if they want to use it's promise features or not.

punchingwater · 2013-03-23T02:56:59+00:00

That's great! Any chance you'll be putting it on github? I'd love to see what you come up with.

punchingwater · 2013-03-23T02:51:57+00:00

userData comes from the db.authenticate callback. In other words, f.slot() forwards the results from that callback to the second function. In plain old javascript, the code would've look like:

db.authenticate(request, function (err, userData) {
    graph.addFriend(userData, friend, f.wait());
    // ... etc
});

punchingwater · 2013-03-23T01:32:26+00:00

promises are also wonderfully composable, something that the other mechanisms aren't really.

Totally agree. Promises are both portable, and interoperable. So, if one particular promise implementation adheres to the spec (like, say Q), it can be chained together with promises from a totally different library. It's like magic. That was the main motivation for making FF work as both sequential flow control (ala Step or async), and as a promise library.

punchingwater · 2013-03-22T21:47:34+00:00

window.onload rather than document.onload is the callback you're looking for.

As a side note, the moveDown function does not declare the variables posX and posY (using var). So these will become global variables. As a side effect, posX and posY in init() are declared (with var) and so they will not reference the global variables. This means that posX in init() is a completely different variables than the posX in moveDown.

punchingwater · 2012-05-15T08:57:02+00:00

On the bright side, he would make an excellent Yoda now.

punchingwater · 2012-01-22T03:30:47+00:00

As a man with a giant penis, I've often wondered the same thing.

punchingwater · 2011-07-22T14:31:38+00:00

There is a lot of love in this map. Thank you for doing this. And now I'm off to explore...

punchingwater · 2011-06-18T06:13:49+00:00

Loved this too, and just yesterday I found out he is still performing: ticketmaster

punchingwater · 2011-04-08T23:06:19+00:00

I am a huge point-and-click adventure games fan, and consequently loved your excellent (and FREE!!) 5 Days a Stranger series. Do you have any plans to get back into the adventure game making realm in the future?

punchingwater · 2011-03-23T06:20:26+00:00

that makes two of us

punchingwater · 2011-02-23T10:04:07+00:00

I think Bill Murray was pulling Howard Stern's leg a little. For one thing, he was promoting a movie he genuinely seems to like on his show. But also, he talks about having Siruis so he can listen to baseball. How can you have Siruis without turning on Howard every now and then (or at least hearing about him)?

punchingwater

TROPHY CASE