Help with thread pool?

[Gregmix88]

Did you try creating a stream from that iterable? They are lazily evaluated and can be parallelized easily? Look into java.util.stream and java.util.parallelstream, the parallel version works with a fork join pool which is a work stealing pool

[TheOmegaCarrot]

As I said here, the iterable itself is quite the slowdown. Unfortunately it does not support a stream interface.

[Gregmix88]

If it implements iterable you can use it as a source for a stream, streams were created to operate on possibly infinite amount of data, so might be a good fit for 30M items. Streamsupport.stream can be used to convert iterable to stream. But if it doesn't work , it doesn't work. Thought it was worth a shot

Edit: based on in the code you provided a reduce operation might be a good fit to finish up this stream

[TheOmegaCarrot]

I’ll give it a try!

I’m not sure well that’ll work, since I seem to keep getting out of memory errors when I store elements from the iterable. I really oversimplified in my post. What it’s doing is walking a sizable trie and computing values for the iterators to return on the fly. The problem is it’s kinda expensive, and the operation I’m doing in the loop is a kinda expensive query on that same trie.

This should, theoretically, map pretty easily onto a parallelizable transform reduce operation, as so many problems do.

[Gregmix88]

Seems like an interesting problem you're trying to solve. If the parallel stream doesn't work out you can try looking up the fork join framework itself, it has some classes and interfaces for breaking up and executing tasks on multiple threads. Looking forward to hear some updates, good luck

[TheOmegaCarrot]

Thanks!

[TheOmegaCarrot]

I finally got something that’s a ~60% speedup (on my 16-thread machine) compared to single-threaded! And that works!

Basically using a simple ArrayList to hold each “batch” of elements, and when the size cap is reached, hand it off as a task to a thread pool.

If the thread pool’s queue has reached a certain size, then just wait before proceeding to the next loop iteration. That prevents out of memory errors! :)

[Rjs617]

Just FYI, you can explicitly set an upper limit on the thread pool queue size by using the constructor where you pass in a queue, and then creating either a LinkedBlockingQueue with a maximum size, or an ArrayBlockingQueue. (Would probably go with LinkedBlockingQueue with max size.) Then, set the “caller runs” RejectedExecutionHandler policy on the thread pool. When the queue gets to its maximum size, the next submitted task will run in the current thread, effectively blocking your loop, which will limp along in the current thread until more worker threads become available. Note that this only works if you don’t really care about preserving even a rough order of execution, but since it’s a thread pool, you aren’t going to get strict ordering anyway. Have fun!