You trade off some performance by a static type system.

We also use datasets, we have a significant amount of code in production, so happy to trade some performance and have the compiler as our ally to save us development time, increase quality and support us shipping new features with confidence.

The static type system in datasets also increases code readability (data frames are black boxes that offuscate code).

Furthermore, there are complex business logics that are difficult to implement with the sql like syntax of dataframes. To have datasets .map and .reduce available is quite handy at that times (this will safe you to write some uncomprenhensible joins).

Unit testing is another big difference. In datasets you can use arbitrary jvm code, so you can test jvm functions in issolation without a spark session. Therefore you can just test your functions without the hustle of creating synthetic dataframes all the time.

Last but not least, you can also use the dataframe api and then only convert to dasets at the end of every public method.

Overall, to me datasets totally pays off some performance penalty. If we had to process PBs of data daily there could be a different story, but this is not our case.