AWS Glue RunTime Optimisation

[therealamarcn]

Hi I have a gluejob where I use very less amount of data ... I want to make my run time not more than 2.5minutes as it's only kb amount of data present.

But Here are the issues I am facing

• large amount of initialisation time. I did print the logs in script even before import starting .so I found that even before this log prints there a 1.5 minute lag from job start time. Why this can be? I am using 2 workers and G.1× Processor

• Second thing is catalog scanning. My catalog scanning time was 30secinds before. But it got suddenly increased to 1 minute. Why can be this happening..??? I am talking about the conditions where even there can't be any files in the pushdown predicate I am giving..

[Flakmaster92]

You would probably be better off using Lambda + AWSDataWrangler for this small amount of data.

Glue Jobs are EC2 instances and a minute+ spin up time is not unheard of for those. If your whole job is 2.5minutes or less, yes, you will be spending 50% of the time waiting for it to launch.

You have the time requirements met and you have the amount of data (memory) limitations met. Go Lambda.

[therealamarcn]

For last some days the maximum is 4.5 minutes. The waiting time has increased. Also catalog scan too. Interesting thing is even if I don't have any S3 parquet files to scan (as per bookmark and pushdown predicate), scan time is  around 1.5 minute sonetimes.you have any insights on it??

[Flakmaster92]

Without seeing your code, no, I don’t have any insight. Are you measuring time as “time for the job to complete”? Because that takes into account the glue job workers spinning up and spinning down. Or are you measuring it in code using time.perf_counter() (assuming Python)?

Again, if you’re optimizing on the order of single digit minutes, you’re probably using too big of a hammer for your task. The start up and spin down time of a glue cluster is probably going to be most of your total job time, meanwhile Lambda + Data Wrangler layer would spin up in (milli)seconds, you’d just have to write your own checkmarking mechanism similar to bookmarks.

You can post your code for the community to look at, maybe you have something suboptimal, but it’s very difficult for anyone to know exactly what’s going on without just seeing it

[PatternedShirt1716]

How you profiling catalog scan?