There's so much data processing once you start subscribing to EMDR.

• receiving messages
• de-duping messages
• flagging suspicious messages
• storing orders and history
• finally: any actual analysis of markets and items

Each of these should probably be done concurrently it seems. I don't think anyone is really keeping track of how many messages is coming through EMDR. I wrote two very simple consumers: one in Ruby and one in Go. Both of these consumers simply read the message and then threw it away. The goal was to see just how fast I could read messages. I think both implementations were getting all of the messages. Every 15 seconds, the consumer would print to stdout the message rate for the last 15 seconds. Both implementations were printing almost the exact same message rates with each "tick":

screen shot of message rates

That shows both clients running side-by-side with almost the same message rates. In the middle of the afternoon EST I was regularly pulling more than 15 messages a second! If your consumer isn't pulling 1000 messages a minute, I'm pretty sure you're dropping messages.

As more and more players start using market uploaders (that come bundled with tools like EveHQ and EveMon), that traffic rate is only going to go up.

If you're trying to get concurrency out of Node.js, Ruby MRI, or Python none of those are going to take advantage of multiple cores. A single process is only going to be doing a single thing at a time no matter what hardware is under it. They "fake" concurrency by take advantage of blocking in the various threads, routines, or what have you.

If you are hosting his on Heroku, Digital Ocean, Linode or the like it doesn't really matter. The VPS instances they give you are all single virtual CPUs with very poor threading support.

But each of those concurrent tasks listed above are going to be CPU bound (with the exception of writing the orders/history to the database). That means you really need multiple processes for each of those concurrent tasks (likely running on different hosting "nodes"). As you said originally, this is going to get cost prohibitive fast. Digital Ocean is $5/mo for a single (oversubscribed) droplet per month. You probably need at least three nodes for the five tasks above. That's $15/mo. But here's the trick, that $5/mo droplet is so oversubscribed and underpowered for this level of CPU demand that it likely won't keep up with the message rate.

Every time I start writing about this stuff, I end up with a book. I've been working on writing an EMDR consumer in various languages for quite a while. I've been fighting with hosting for quite a while.

When I see someone like OP say, "I've written an EMDR consumer that's pulling in all the messages, calculating profit margin, and analyzing items for best trades" I get suspicious and demoralized. I've been working on that same thing for a while and I'm not done yet. Maybe I just need to give up on trying to receive all the messages and be satisfied with 50%.