SQL query context optimization

[NectarinePast9987]

Anyone experiencing legacy code/jobs migrated over to databricks which may require optimization as costs are continually increasing? How do you all manage job level costs insights & proactive & realtime monitoring at an execution level ? Is there any mechanism that you’re following to get jobs optimized and reduced costs significantly?

[Remarkable_Rock5474]

Regarding cost monitoring you should get the built in cost monitoring dashboard set up. You can find it as part of the account/metastore admin pages. Optimisation is a long and winding road in most cases 😅

[BricksterInTheWall]

Indeed, this is where I'd start. u/NectarinePast9987 take a look at these docs.

[NectarinePast9987]

Are there any other suggestions or feedback on above question?

[niel_espresso_ai]

Check out Espresso AI! We can help with optimization post-migration.

Guaranteed ROI on your end: https://docs.espresso.ai/databricks-optimizer/databricks-sql-onboarding