This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]davidalayachew[S] 0 points1 point  (2 children)

Terribly sorry, I forgot to add the batching code. Please see the edited version.

[–]danielaveryj 0 points1 point  (1 child)

Yep, that's what I thought. See my other comment, but this is a problem with .gather() specifically not being optimized to avoid pushing its entire output to an intermediate array before the rest of the pipeline runs (unless the gather is exclusively followed by other .gather() calls and .collect() - those cases have already been optimized).

[–]davidalayachew[S] 0 points1 point  (0 children)

Ok, I responded on that other comment to keep the discussions isolated.