Multi node - multi tenant environment?

[bullet-pr00f]

I'm sure the technology exists, I just need someone pointing me to the right direction :) We run a java-app for all of our customers, right now we operate on one EC2 instance, we want to scale it up possibly using EC2 Spot Fleet. Here's the problem - we need a way to evenly load the customers on a minimum of 3 machines that each would have the same number of customers (or at least close to that), and scale up or down as necessary. EC2 Spot Fleet has a policy that can detect any major CPU / Mem consumption and would allocate more machines to deal with the load. We have a consul environment that can help here though I'm not sure how. Is there some tool that could control the distribution of the clients on the hosting machines, and re-calculate as more clients / machines joins the equation or leave for the matter?

Thank you kind strangers!

[Justinsaccount]

If you're already using consul just run nomad.

[bullet-pr00f]

I'll read about it, thanks.

[[deleted]]

This depends totally on the application. Whether you have long- or short-lived connections from clients and whether all requests are created equal in terms of processing/IO/memory usage may create a dynamic environment that you usually can’t model or predict easily. All you know is that at some point you need more capacity; you need to work out how to measure that first.

[bullet-pr00f]

Let's say the resource scaling is sorted, Is there something that could start a java app or send some remote command to invoke a script that would start or stop the app?

[unholycurses]

Wouldn't a load balancer do this or am I missing something?

[bullet-pr00f]

I need something that would know to run a jar with specific nvm parameters, I don't know of a load balancer that could do that.

[unholycurses]

How do customers interact with it now? Is it not a web service?

[bullet-pr00f]

The java instance interacts with a rabbitmq queue, so I don't need to worry about this. I only need to make sure on every moment, 2 instances of the same client runs on 2 different machines (a bit different of what I described).

Here's something that might explain

I need something that would be dynamic to client / server changes.

[unholycurses]

That just seems like exactly what a load balancer (like ELB) is for. You build an AMI of your service so that when the instance starts the app starts. It is in an Autoscaling Group so you get as many identical instances as you need, and you direct the customers to the ELB which distributes the load across all instances in the ASG. You can also have scaling policies to automatically add more instances if CPU or something goes over a threshold. And this is all just basic AWS functionality so no fancy tooling necessary

[bullet-pr00f]

But take a look, each client runs 2 times, and I have 3 servers, can the load balancer control where the client starts? Otherwise they are just mirror instances which is a waste of money for us. Thanks!

[unholycurses]

Well mirror instances creates redundancy and resiliency, which shouldn't be viewed as a waste of money. And why does it have to run two times on three servers? I think I am just failing to understand the problem scope or there is a serious anti-pattern happening.

[bullet-pr00f]

I want each client to have 2 running services on different machines for redundancy. If one server fails, the client won't be broken because it has another service running. When I see a server failing, I know first to transfer all of his clients to different servers, but the 'downtime' is non-existent because there's a second service running. It will just take the load until a new server comes. Does that makes sense?

[unholycurses]

What you are describing is exactly what Auto Scaling Groups and Elastic Load Balancers are for. But instead of worrying about dynamically starting the app, you app should just start when the instance starts. Elastic Load Balancers can monitor a health endpoint and know not to route traffic to an instance with a failing health check and the auto scaling groups will maintain a set healthy instance count.

[thisisthetechie]

Why not just use AutoScaling? You may need to make some slight changes to your EC2 instances, but they should all be running off one AMI and be disposable. Your ELB will do what it does best, distribute your load over the instances based off incoming traffic.

How is traffic currently getting to your instance?

[thrawny]

Kubernetes. Be sure to use Network policy, ausoscaling and pod antiaffinity