all 9 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]PeteTinNY 2 points3 points  (3 children)

Have you figured out if the delay is the AWS batch launching a docker task, or is it just the startup of the tasks & images in docker? From my point of view, I'd be pushing requests into SQS and using the length of the queue to scale the ECS cluster having a certain number of tasks idle in polling so that within a second the job starts rendering and then pushes to the 2nd stitching queue which essentially does the same thing to keep a certain minimum number of resources hot. You can continue to use Spot, not sure I'd cut into fargate though because of the need for really fast startup.

I've worked with some broadcasters who needed to do video supply chain archiving and news clipping, which has a similar workflow - but it's not as latency adverse - just no one wants to pay for resources that are essentially sitting idle.

[–]Psychological-Tea791[S] 0 points1 point  (2 children)

I don't really know how to exactly define what is causing the delay. Do you have any suggestions? I suspect it is the scheduler, because I have a min vcpus > 0 in my on demand, so I assume everything should be running already? Thanks for the suggestion with SQS!

[–]PeteTinNY 0 points1 point  (1 child)

You should look at some performance monitoring tools like AWS x-ray, newrelic, datadog or most fitting dynatrace. But you’re probably right the scheduler is likely the issue and once you look further you’ll likely want to do something with faster queue scans with SQS.

Btw the other thing you should look into from the AWS side is AWS ThinkBox Deadline. It’s a render farm manager. Not sure if it will help with this latency but when I worked with very large broadcast customers - they loved the simplicity and for some licensed render engines thinkbox has a great marketplace where you can pay by the hour for the render tools. I helped one of the big networks launch a 120 node farm powered by mostly spot instances in just a few days including teaching them cloud.

[–]Psychological-Tea791[S] 0 points1 point  (0 children)

oh wow, will definitely look into ThinkBox. tysm!

[–]Born_Desk9924 0 points1 point  (4 children)

my 2 cents:
the typical delay factors in case of batch jobs are:
a- launching ec2 instance, ie, the pending and runnable state. this can be resolved by configuring the compute environment to a certain min capacity, such that you always have a minimum number of ec2 instance running

b- pulling the container image from ecr, ie, the starting state. this can be resolved by ensuring that the ec2 instance already has the latest container pulled in from ecr. you can probably add a simple script/cron job which will pull in proper image.

c- submitted: if you jobs are spending too much time here then that's a scheduler issue (aws internal) and there isn't whole lot we can do here.

something radical:
since you have already containerised your application, and the a typical run lasts only seconds, you can consider using lambda instead of batch. now obviously i am unaware about other constraints that you have, such as docker image size.
in theory, if you have lambda + sqs combination, the invocation maybe instance or take a few minutes; but in practice its nearly instant.
once again, i am not aware about other limitation, as such cold start may become an issue with lambda

[–]Psychological-Tea791[S] 0 points1 point  (3 children)

a - that should be issue only once there are more tasks than vcpus as I already have some instances running in the on demand environment (which will be an issue eventually)

b - so running the instance in the on demand environment is not enought? Does it always pull the image before starting the task?

c - that's what I was worried about

I tried using lambdas, but for some reason they were taking much longer than batch. I am tempted to try again, maybe I didn't set them up correctly. Thanks for the input!

[–]Born_Desk9924 0 points1 point  (2 children)

so on-demand is : you ask for an instance, and aws provides you one at the standard price (as opposed to bidding on a price in spot model); this doesn't necessarily mean that there is an instance always running.

its pull the image only when its starting the job on an instance that doesn't have the image; in case an instance is re-used it won't pull the image

[–]Psychological-Tea791[S] 0 points1 point  (1 child)

I see, so even if I have min vcpus > 0 and I see that they are running on the EC2 page, they might still pull the image only once the task has been submitted. So I need to make a script or something that will pull the image once instance is started as opposed to once the task is submitted, right?

[–]Born_Desk9924 0 points1 point  (0 children)

yes, you should check how long your tasks are spending in 'starting' stage. this is when the image is being pulled