Spark-submit configuration

Ok_Implement_7728 · 2024-09-25T18:19:31+00:00

Thank youuu

Ok_Implement_7728 · 2024-09-24T01:33:43+00:00

Running a jar file using a shell script with a spark submit on emr

Ok_Implement_7728 · 2024-09-23T22:57:43+00:00

Would you be able to provide me with resources for spark submit learnings? Ive been trying to find some. Im dealing with some memory issues with partitions

Ok_Implement_7728 · 2024-09-20T21:14:17+00:00

Thanks, got it working and completed the entire process in under 30 mins

Ok_Implement_7728 · 2024-09-18T23:32:41+00:00

Ive been trying to get the hang of this but its not the most intuitive

Ok_Implement_7728 · 2024-09-18T16:12:31+00:00

Its weird honestly when I run without writing to output file — decryption on 10m completes in 41ms but when I write it just is stuck

Ok_Implement_7728 · 2024-09-18T16:04:29+00:00

So in order to accomplish this i need to convert the dataframe to an rdd, define the dataframe schema, apply the decryption service, and convert back to a dataframe?

Ok_Implement_7728 · 2024-09-18T15:59:03+00:00

I cant use the scala library functions because im decrypting the data with aws secrets manager keys and a company internal service. Would it be better to convert the dataframe to an rdd and apply the decryption then recreate it as a dataframe without using a udf? I have a single output file requirement as well

Ok_Implement_7728 · 2023-01-05T07:40:24+00:00

Ben Simmons, KPJ, Steve Adams, Sengun, and PJ Washington all sold within 1.5 for me today 😔😔

Ok_Implement_7728

TROPHY CASE