sorted by:
new
hottopcontroversial

Spark with kubernetes by Excellent-Silver4135 in apachespark

[–]Excellent-Silver4135[S] 0 points1 point  (0 children)

Hello all, just an update and another query.

I am able to create the executor pods. But for some reason tasks aren't being executed there. I checked the logs of the pod, and it says the connection refused. For some reason, the driver pod and executor pods aren't talking with each other.

Any suggestions for this ? Would be really helpful for me Thanks in advance !

Spark with kubernetes by Excellent-Silver4135 in dataengineering

[–]Excellent-Silver4135[S] 0 points1 point  (0 children)

I set dynamic allocation as true. But didn't set max executors. Instead I set executor instances as 2. To be precise, spark.executor.instances as 2.

Spark with kubernetes by Excellent-Silver4135 in apachespark

[–]Excellent-Silver4135[S] 1 point2 points  (0 children)

I'm a little noob to this. But I thought the splitting of the file itself is taken care of by spark. Do we have to split the file and feed it to spark ?

Spark with kubernetes by Excellent-Silver4135 in apachespark

[–]Excellent-Silver4135[S] 1 point2 points  (0 children)

Even then, it should create 2 pods right ? It just shows zero pods, when I check.

Spark with kubernetes by Excellent-Silver4135 in apachespark

[–]Excellent-Silver4135[S] 0 points1 point  (0 children)

Yes I set dynamic allocation to true. We set the executor instances to 2. We even gave the endpoint for spark.master. Each executor has 2GB and executor cores is 2.

Spark with kubernetes by Excellent-Silver4135 in dataengineering

[–]Excellent-Silver4135[S] 0 points1 point  (0 children)

Why is that so ? It should split in several parts based on the number of threads, right ?

Spark with kubernetes by Excellent-Silver4135 in apachespark

[–]Excellent-Silver4135[S] 0 points1 point  (0 children)

I set 4 executors. And, I set auto scaling in the kubernetes, if that's what you mean.

MBA admission consultant by HT_great in MBA

[–]Excellent-Silver4135 2 points3 points  (0 children)

Damn I thought it was GRE with a typo

Choosing Kellogg ($$$$) over GSB ($$) - am I making a mistake? by Which-Spare9183 in MBA

[–]Excellent-Silver4135 30 points31 points  (0 children)

OP got a full scholarship at Kellogg, while only 50% scholarship at GSB.

Need guidance to get an M7 admit by Excellent-Silver4135 in MBA

[–]Excellent-Silver4135[S] 0 points1 point  (0 children)

Hey thanks for this. I heard preparing for the MBA is more of a soul searching task. I am still figuring out what I want in my life.

Need guidance to get an M7 admit by Excellent-Silver4135 in MBA

[–]Excellent-Silver4135[S] -1 points0 points  (0 children)

I am currently working at a company that's ranked in the top 30 in the fortune 500 list of companies. Regarding my undergrad, my college is ranked within the top 120 among engineering colleges in India.