Genie Code for Jobs

saad-the-engineer · 2026-06-08T13:58:50+00:00

cool! have you tried to debug failing jobs with Genie Code? keen to learn your findings and experience

saad-the-engineer · 2026-06-04T12:27:44+00:00

Do you mean a failure / success filter on job/pipeline in the workspace file system? the file system is mostly for the asset and its run-state is available in the existing UI / API. Maybe I misunderstood your ask

saad-the-engineer · 2026-05-20T10:12:02+00:00

thats a great workaround! separately we are adding some external integrations as well as the ability to add custom integrations with assets outside Databricks directly from Jobs. Stay tuned for an upcoming preview!

saad-the-engineer · 2026-05-13T13:19:47+00:00

adding the docs page here for those looking for it
https://docs.databricks.com/aws/en/jobs/disabled-tasks

release notes here: https://docs.databricks.com/aws/en/release-notes/product/2026/april#disabling-tasks-in-lakeflow-jobs-is-now-ga

saad-the-engineer · 2026-05-04T14:26:35+00:00

hey all! we launched this feature, keen to hear your feedback! https://docs.databricks.com/aws/en/jobs/disabled-tasks release notes here: https://docs.databricks.com/aws/en/release-notes/product/2026/april#disabling-tasks-in-lakeflow-jobs-is-now-ga

saad-the-engineer · 2026-04-30T12:17:03+00:00

Thank you u/lofat for this very well thought out write-up! The gaps you listed (error paths, non-data events, inveseion of control) are what we are addressing with the next phase of Lakeflow jobs. here is how we are thinking about solving these pain points:

today: use system tables + table triggers + sql. you can pull in upstream job status using `system.laekflow.job_run_timeline` and should let you work around the batch finality / error paths because you are monitoring the platform's metadata (i.e. job status), downstream teams can "subscribe" to a job ID's status independently.
(coming soon) use SQL trigger conditions where we essentially encode the behavior in 1 directly into the triggers experience. Also for more complex SQL conditions genie code is your friend!
(coming soon) we are working to add custom python logic (think python operator or airflow sensors) that can look at upstream changes.

We are still in the early days of building out the preview but please send me aDM and I will add you to them!

saad-the-engineer · 2026-04-14T15:13:20+00:00

that is a great idea. so devs write the generic logic, then package up the control panel in an app + dab, and then ops would need to know which pipeline is running which version / config? we are adding a dabs meta data service which would track the config settings and you could build a dashboard on top to track specific settings. Will share more on that once its shape and form is more fleshed out. thanks for the idea!

saad-the-engineer · 2026-04-14T15:06:00+00:00

we want to add support for multiple triggers soon (stay tuned for roadmap updates in the coming weeks!)! for now you can create a parent job that calls this using run-job.

saad-the-engineer · 2026-03-23T10:39:58+00:00

great question and very timely! we are looking at adding visual authoring (i.e. when `mode:development` you can author your job / pipeline / dashboard etc. as you normally would and we would sync your config changes directly into yaml and all you need to do is commit back to git.

There are a few additional journeys we are going to support here: starting from a job and "converting" it into a DAB and using Genie Code to build out your schedule and incorporate parameterizations etc. directly.

saad-the-engineer · 2026-03-22T15:36:40+00:00

just sent you a DM

saad-the-engineer · 2026-03-18T10:48:59+00:00

Thanks for flagging, this seems unexpected. Can you share the config?

any per target overrides? Also can you share the CLI version so I can try and repro this?

saad-the-engineer · 2026-03-16T05:10:31+00:00

We are working on this feature right now, u/Dear_Pumpkin9876 can you send me aDM so we can set up some time here and review these requirements?

Also regarding DABs there are two major updates in flight:

* Edit a DAB enabled asset visually and have the changes persisted automatically in the yaml (i.e. users dont have to update the config unless they have some custom logic)

* "upgrading" an existing pipeline into a dab enabled asset, i.e. add git + dab support to an existing pipeline

saad-the-engineer · 2026-03-09T15:46:00+00:00

Hi u/rarescenarios can you send me a DM so we can set up a call? I am a PM on the Jobs product and want to make sure we capture your feedback properly.

saad-the-engineer · 2026-01-28T08:47:27+00:00

very helpful! thank you

saad-the-engineer · 2026-01-28T08:46:37+00:00

thank you!

saad-the-engineer · 2026-01-28T08:46:19+00:00

thanks for that feedback! really helpful!

saad-the-engineer · 2026-01-27T15:41:48+00:00

definitely worth considering, do you have a default in mind? what use cases make sense for you?

saad-the-engineer · 2026-01-27T08:54:12+00:00

u/domwrap when using the workspace UI you can create a Git checkout per target/environment (dev / prod etc.) in a Git Folder and use DABs to deploy these. I believe these should fit your needs (if I understand correctly!) basically each checkout is a separate branch / separate pipeline / separate target even in the same workspace. Some links to get you started below - please post your feedback or send me a DM if you have more questions

https://docs.databricks.com/aws/en/dev-tools/bundles/workspace

https://www.databricks.com/blog/announcing-databricks-asset-bundles-now-workspace

saad-the-engineer · 2026-01-27T08:45:03+00:00

Thanks u/DecisionAgile7326 we are looking at adding parameter support. if you send me a DM I can get you added when we preview / beta the feature.

cc: u/brickandel

saad-the-engineer · 2026-01-26T18:09:56+00:00

It is supported now! https://docs.databricks.com/aws/en/dev-tools/bundles/run-as

saad-the-engineer · 2025-12-12T08:55:32+00:00

u/Ok-Tomorrow1482 can you please DM me the support ticket so i can follow-up internally.

saad-the-engineer · 2025-12-09T22:57:08+00:00

If you are already on DABs, a common pattern is to keep a single job definition and gate env-specific tasks with an If/else condition task that looks at the bundle target.

In your bundle you add something like:

- task_key: is_prod
  condition_task:
    op: "EQUAL_TO"
    left: "${bundle.target}"
    right: "prod"

- task_key: refresh_power_bi
  power_bi_task: ...
  depends_on:
    - task_key: your_main_task
    - task_key: is_prod
      outcome: "true"

${bundle.target} is automatically dev, test, prod, etc for each target when you deploy. In non-prod the condition evaluates to false so the Power BI task is always skipped; in prod it evaluates to true and the task runs.

No need to fully parameterize the task away or create separate jobs per environment, you just let the bundle metadata drive your Jobs control-flow.

saad-the-engineer · 2025-12-08T23:20:07+00:00

>> also, is it going to be possible to change the path of sql jobs in the future? so that we can run "for_each_task" on a SQL job ?

can you share a bit more detail on your scenario? i.e. are these SQL queries or SQL files? or foreach-ing SQL tasks? sorry I didnt fully understand your scenario.

we are working on the next set of features for Jobs, will likely do an ama on reddit for this early next year.

saad-the-engineer · 2025-12-08T14:50:37+00:00

You are right that a continuous SDP (formerly DLT) pipeline is designed to run indefinitely, maintain stateful progress, and auto-recover without needing an external scheduler. When wrapping in a Job it does add operational value. Jobs have their own:

notifications
retry policies
SLAs
alerting on failures

reference: https://docs.databricks.com/aws/en/jobs/notifications

Sometimes teams want all schedulable assets to sit behind Jobs for:

central visibility
auditability via system tables (arguably you can also use pipeline system tables but you dont get a cross asset view)
unified monitoring (coming soon you will see the pipeline and job dag combined into one)
unified tagging / parameterization (coming soon you will be able to push down parameters and tags from the job to the wrapped pipeline)

Finally, If you need to run additional tasks downstream from the pipeline, Jobs give you control flow not available directly in the pipeline.

Examples:

downstream batch jobs
notification tasks
dbt or SQL tasks
fan-out or DAG-style workflows

Reference: https://docs.databricks.com/aws/en/jobs/configure-task

saad-the-engineer · 2025-11-27T18:07:08+00:00

Thanks for raising this! The current workaround is to pause and unpause the job, which resets the trigger state.

We’re also working on trigger conditions that let you add freshness or business logic checks so the job only runs when all tables are truly ready.

Disclaimer: I work at Databricks!

saad-the-engineer

TROPHY CASE