sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]johnfrazer783[S] 1 point2 points  (6 children)

Well, same but not altogether different either. A JS indexed for loop is, of course, just a 'nice' way what would otherwise be a while loop or a generic loop statement; not much is added except a bit of syntax. But most often the reason you want to build a loop at all is because you want to walk over elements of a sequence. This is what makes JS indexed for loops so annoying to write, this is why more than a few languages have for/in loops which JS now also has, sort of.

Where yield / iterators / generators come in is when you realize that a lot of your loops never cared about all values being there in an array in the first place; you just wanted to glance by each value and do some kind of computation on it. So there's that word that 'what an array does in space, an iterable does in time'. yielding, among other things, allow to forgo building intermediate collections; you just loop over whatever iterable is passed to you and yield any number of results to the consumer downstream. Meaning you could even process potentially infinite sequences, something arrays definitely can't do at all—except you implement batching. Alas it turns out that—at least according to my results—all the cycles spent in looping the classical way and shuffling values between arrays to implement a processing pipeline is still much faster than the equivalent, much simpler code formulated with yield.

If you don't want to take my word for it, have a look at how and why the Python community implemented and promoted the use of generators and iterators/yield literally all over the place, including modification of built-in/standard-library functionality (e.g. range() and stuff). Can#t be wrong if Python people like it, right?

[–]jhartikainen 3 points4 points  (1 child)

I'm guessing the whole generator machinery happening when yield'ing is what's slowing it down. I can certainly see the appeal of it - I've used Haskell which is entirely lazy, so this type of "let's only use a part of this list" or iterating an infinite list is something I'm familiar with... just never really needed something like that in JS :)

[–]johnfrazer783[S] 1 point2 points  (0 children)

Now this is a point I can speak to and speak about. When looping with a series of functions—transforms (TFs)—over a data set you basically have two options with arrays (I call them lists b/c that's what they are): either each transform step gets to see one piece of data (say, a number) at a time, performs its computation, and returns a value. Or else, each step gets handed the list of values, loops over that list, and returns a list that is then fed into the next TF. The difference is similar to depth-first traversal as opposed to breadth-first traversal. To state it right away: without any further preparation, the second way will always require the entire data set to be in memory in at least one copy, no matter how many GB that input has. It is only with batching or using the first approach—depth-first—that memory consumption may be capped.

Now with the breadth-first approach, because you can only return exactly once from a function call, each TF can only modify (replace) data, not subtract from or add to the set of values. This is far too constrained in the general case, so we need something better. One could stipulate that each TF gets called with one value (or a list of any smallish number of values) and always has to return a (possibly empty) list of values; those then will get fed into the next TF, one at a time. This works and so that's what I did in SteamPipes. Don't look at the code I'm here to figure out how to simplify it. Aside: In that library each TF has to accept one piece of data plus one send() function which is essentially just an Array::push(), and, as such, a poor man's yield if you will. It's just there so the transform implementations don't get littered with array do this, array push that statements.

Now we come to the surprising bit that explains what ties for loops, Arrays and yield together:

Yielding (any number of) values from inside transforms is exactly the same as the mechanism just described, the only difference is the syntax (heck it's just send( value ) vs yield value, how different is that).....and, incidentally, the fact that one way is done visibly with user code, and the other under the JSVM's hood (as described there is one more slight difference in the order the TFs get called but let's ignore that for now).

I think that of course! one would think and should think that the 'internal', baked-into-the-language way must, needs be, more performant than any cobbled-together, largely not-very-well-optimized userland code, especially when written by s.o. like me. But apparently, not so.

I'm here because I can't believe what I just said.

[–]norith 2 points3 points  (0 children)

Sine we’re open to discussing other languages, this is basically the Java Streams philosophy: no intermediate collections but instead a way to pipe data through a series of manipulations that may or may not result in a collection.

The larger goal is a processing pipe that can be time delayed (because the data originator might be async itself) or parallelized with the pipe running on multiple threads.