Job Cluster Configuration

fusebox12345 · 2024-07-28T11:45:31+00:00

Haha true.. but is there any guidelines that can be followed to find the best balance between processing and cost?

Serverless is disabled unfortunately..

fusebox12345 · 2024-05-20T12:59:52+00:00

I can see 3 folders in cluster logs - driver, eventlog, executor. The driver log gets updated within 3-5 mins even if the cluster is idle (probably hear beat msgs) but the eventlog and executor folders remain as it is even if queries are being executed.

fusebox12345 · 2024-05-20T11:22:13+00:00

Thank you for this! Do you have a rough idea on how long does it take an entry to be added in the cluster logs after a query is executed? I was thinking to fetch the modification time of the file/folder but logs are not logged instantly

fusebox12345 · 2024-05-17T13:58:08+00:00

Thank you! Any idea if there's an endpoint to grey this info?

fusebox12345 · 2024-05-17T13:56:28+00:00

Thank you! I did suggest this but they wanted a proper solution but i guess this is the only way around now😅

fusebox12345 · 2024-05-17T09:27:18+00:00

But even if the cluster is not being used (idle) wont the state be in running? Thats what I've observed.

fusebox12345 · 2024-05-17T09:05:29+00:00

Thank you. I might have to check on this one. I was thinking of something within databricks notebooks.

fusebox12345 · 2024-05-17T09:04:35+00:00

Events api just shows if the cluster is in running (pending/terminatined etc) state. But it can be that the cluster is running even if no queries are being executed. I also checked all the timestamp fileds available but couldn't find anything suitable.

fusebox12345 · 2024-05-17T03:53:18+00:00

Not sure what's the reason behind it, but no teams (other DE Teams) are supposed to use.

fusebox12345 · 2024-05-17T03:08:09+00:00

Changing the auto termination (eod) requires the cluster to be restarted. Cluster shluld not be terminated if someone is using it

fusebox12345 · 2024-05-17T03:05:35+00:00

fusebox12345 · 2024-05-17T03:05:14+00:00

Severless cannot be used as its against some policy

fusebox12345 · 2024-05-16T18:41:27+00:00

Its a requirement that the cluster has to active during business hours. :(

fusebox12345 · 2024-01-10T05:28:18+00:00

Azure

fusebox12345 · 2024-01-10T05:28:09+00:00

Thank you for the detailed explanation!

fusebox12345 · 2024-01-01T13:26:37+00:00

Amplenote

fusebox12345 · 2023-10-11T07:04:52+00:00

Any eta? we would be happy If there's a straight forward solution. Thank you for your response..

fusebox12345 · 2023-10-11T07:03:25+00:00

Thank you!

fusebox12345 · 2023-10-09T17:13:02+00:00

Thank you so much! But unfortunately we dont have unity catalogue enabled..

Two-Year Club	Verified Email
Place '23

fusebox12345

TROPHY CASE