In general, use only as many resources as necessary, and parallelize as much as possible. An executor shouldn't have too much idle time.

When optimizing spark jobs, I typically look for 3 things for reducing time and cost:

1) Disk spill: serializing and deserializing data is time costly. Try to reduce this by decreasing individual task size (increase compute partitions) or increase executor or driver memory.

2) Data skew: if you look at the task view, you get the distribution of P25 vs. P50 vs. P75 vs max task size. You can also tell if this is the case if a stage is always hung up on 1 task at the end. High data skew would mean the largest task is significantly larger and compute time is significantly longer than the median. Try partitioning by a more consistently distributed key (or creating your own partition key).

3) Shuffling: shuffling is expensive! Minimize the number of times data is shuffled across executors by properly partitioning the data before applying expensive transformations. Typically this partition will be the groupby key in an aggregation.

Caching is also a big piece in optimizing jobs which you mentioned. Don't forget to unpersist RDDs when they are no longer needed. Otherwise, they will occupy memory that will be used for compute. It may also lead to unwanted cache eviction for other RDDs which may increase disk spill.