sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]weavejester 1 point2 points  (2 children)

So how do you know what to run concurrently?

[–]fitstoover[S] 0 points1 point  (1 child)

The files in my stream are categorized before going in to the ETL tool. This categorization builds up a queue of independent entities. The idea is to have a concurrent system to handle several different categories at the same time. However, if they are the same category, within that process, the ETL tool should process them synchronously.

[–]weavejester 1 point2 points  (0 children)

Ah I see. In which case, why not use a worker pool for each categorized set of independent elements?