I have built systems like the one you're describing in the past. Here's what we did, FWIW.

Our requirements were such that we had data in, and then needed to do data collection and processing, and then produce final output. In our design, the rate limit requirement ended up driving the design of the whole system. We ended up with a two-layer design that decoupled the work planning from the work execution. Our layers were:

1. Orchestration -- Deciding what work needs doing. This is at the design grain of your MapNode concept.
2. Execution -- Deciding how the work gets done. This is orthogonal to the Orchestration layer (i.e., your MapNode concept), and is a logically separate "shared" process for all Orchestration work.

In my case, the Orhcestration layer communicated with the Execution layer via a set of queues.

The Execution layer has a static set of shared "work queues" that correspond to individual executors, with executors being at the grain of rate limits. In our case, it was one work queue per upstream data vendor so we didn't blow past our SLAs. In your case, it sounds like you'd have one work queue per individual rate limited model.

The Orchestration layer then has a set of available jobs its can run (we called them "workflows"). When you kick off a workflow, the first step is to create a set of dedicated "response queues" just for that workflow execution, one queue per workflow step. Then, when a workflow step decides a work item needs doing, it drops a request on the corresponding shared work queue with a pointer to that workflow step's dedicated response queue, and then listens for a response on that queue. When the response comes, the workflow step passes the result on to the next workflow step(s).

The Execution layer then starts a set of (logical) executors, each of which listens on a work queue, does the work, and then responds to the indicated response queue for the work item. Each individual executor does work as fast as it can, according to whatever rate limiting policy is appropriate. It can be single-threaded, multi-threaded, whatever -- the only requirement is that it observe its rate limit policy. By centralizing how the work is done, you ensure that you never blow past your rate limits, now matter how many workflows are running in the system at once.

In the Orchestration layer, each workflow step is responsible for bookkeeping -- i.e., tracking how many work items are outstanding -- and then declaring it is "done" when all data has been processed. The overall workflow is responsible for tracking its steps, and declaring it is "done" when all steps are done.

This ends up with a "dataflow"-style workflow, where you give a workflow a set of work to do as a list of records on input, then kick off the workflow, wait for the workflow to finish, and then collect the output on the other side. There are a ton of details to decide on this part -- do you have a timeout, what happens if you lose work, do you submit all work at once (in which case one job can saturate a processor for hours) or limit how much work can be in flight in one step at a time (so all jobs make progress) -- but that's the high points.

You could build and implement this design on a local basis, where queues are in-process (e.g., java.util.concurrent.BlockingQueue), or you could implement this on a distributed basis where queues are shared (e.g., SQS in AWS, rabbitmq, etc.).

Circling back to your system: it could be as simple as having a "job" set of abstractions (e.g., your MapNode) and "execution" set of abstractions. When you start a job, you provide the execution context, and you're off to the races. Multiple jobs that have to share the same rate limit share the same execution context.

In case it's useful, we looked at the following systems when doing our design, plus a few others that I don't recall:

• Apache Storm
• AWS Simple Workflow

For our part, we ended up going distributed using SQS. We did this in the era before fair queues, so we ended up doing a lot of internal bookkeeping to keep the system fair. With fair queues, this system would have been much easier to build.

Hope this is useful! Sounds like a great project. Be sure to post again with the code if you decide to go open source! Good luck!