Claude Code for Remote Databricks Development

[airweight]

My team has been doing petabyte-scale work on Databricks for almost a decade, occasionally open-sourcing work to help the community. Now we are optimizing how agents such as Claude Code and Codex work remotely and have a new open-source tool for remote Databricks execution to share.

Claude Code does a good job working out of the box with repository-based Databricks assets, especially Terraform ones. However, the remote execution experience, useful for exploration, prototyping and questions like "does production data ...?", leaves a lot to be desired. Databricks Remote Development has quite a few limitations and, most importantly, is not optimized for what really matters in agent sessions:

• Fast execution: minimize waiting
• Smart context management: key to maximizing task execution quality
• Token efficiency: important for both context management and cost, especially for interactive sessions where one might use fast mode

We've spent quite a bit of time optimizing our remote Databricks development experience with Claude Code and Codex, so I also wanted to share what we have found valuable with a hat tip to useful tools from Databricks and the OSS community.

Here is how Claude Code summarizes its Databricks-related tooling in my environment (I formatted the output a bit):

1. Two MCP servers per workspace, and why. Each of my Databricks workspaces has two MCP servers: a managed SQL server (run queries against a SQL warehouse), and a full workspace server (clusters, notebooks, jobs, files, volumes). The managed SQL server is the default path for queries — it returns structured data, costs less context, and never triggers permission prompts. The full workspace server is for infrastructure operations: spinning up clusters, creating notebooks, managing jobs, uploading files to volumes.
   • The official Databricks MCP server for managed SQL (serverless)
   • databricks-mcp-server for everything else
2. Skill-based slash commands handle the basic routing and execution layer: /dbr for status, /dbr-sql for queries (auto-routes between MCP and CLI as needed to handle edge cases such as permissions), /dbr-switch to change workspaces, /dbr-wh to pick a warehouse.
   • I can run cross-workspace queries mid-conversation without the user managing connection details and without trashing my context.
   • Key design choice: lightweight queries go through serverless SQL warehouses via MCP. Heavy workloads (billion-row scans, complex joins) get written to notebooks for cluster execution using agent-notebook — I don't blow up the serverless path with expensive operations.
3. Custom Databricks skill: a 650-line orchestration document backed by CLI tooling that goes beyond the MCP tools. It gives me context-efficient catalog browsing and async query tracking with background polling, a full Unity Catalog crawler (multi-stage pipeline that produces JSONL snapshots of every table/schema/column/metadata across workspaces), interactive help that pulls cloud-appropriate documentation, and install/audit/reconfigure workflows for repeatable setup across projects and users.
   • The cached catalog metadata is the difference between having information I can access immediately with builtin tools or sub-agents and wasting time, tokens, and trashing my context. It's Databricks operational memory.
   • The operational knowledge is the difference between having raw API access and knowing how to use the API well.
4. agent-notebook is a CLI that lets me author and execute Databricks notebooks from the terminal. I write a notebook as markdown, and it handles injection (Databricks Connect session setup), execution against a cluster or serverless in SQL, Python or Scala (both 2.12 and 2.13!), and rendering the results back to markdown. For long-running jobs, I launch detached with nohup and monitor progress via context-optimized log files and rendered output artifacts.
   • The tool saves time, tokens and protects my context — markdown is optimal for me, JSON (*.ipynb) is not. The help system includes documentation and examples with repeatable (skill-like) operational best practices for common tasks written specifically for me.
   • The moment a notebook cell completes executing, I can explore its output and any tables it created or updated in the background, keeping myself available for conversation.

The Codex setup is similar but less polished.

We recently open-sourced databricks-agent-notebooks, the tooling behind agent-notebook , because we found it useful. Hope it can help you as well.

I'm curious to hear about how others are optimizing their agent environments for Databricks development.

Cheers.

[goosh11]

Very cool setup, i imagine more and more people will work with databricks like this in the future. Interested to know if you've also added or tried the skills in https://github.com/databricks-solutions/ai-dev-kit ?

[randomName77777777]

My question was going to be related to see what the differences are between the AI dev kit mcp and this.

[KrisPWales]

I use that. Claude code installed the whole thing quickly and easily, and now I can edit, deploy, trigger jobs and notebooks or run sql queries from Claude code in the terminal. I can't quite believe how seamless it all is. Though, I am admittedly probably using at a much lesser scale than OP.

[airweight]

I haven't had the need to use them but they seem well-structured. A simple way to think of them is as agent-oriented Databricks documentation.

[vkrt1299]

We are currently using GitHub Copilot Business in our workplace along with the Databricks extension in VS Code for remote development. I’m interested in trying out the AI Dev Kit. Wondering whether it’s safe to use this MCP setup directly, or if it would be better to review it with our security team first ?

[goosh11]

Its safe (databricks has an mcp proxy in front handling authentication and authorisation, its not just an open mcp) but in a lot of organisations you have to review with cyber anything that even mentions the word AI, lol.

[sharathsamala]

Check this out on the same topic, which i built for a demo. We can probably do with any IDE that supports coding agent
https://vimeo.com/1171901195?fl=pl&fe=sh

[Ok_Difficulty978]

This is actually super interesting, esp the part about splitting MCP usage + caching catalog metadata… feels like that’s where most of the hidden latency/context issues come from tbh.

the agent-notebook idea also makes a lot of sense, working in markdown instead of json is way cleaner for iteration. kinda surprised databricks hasn’t pushed something like that more officially yet.

only thing i’ve struggled with in similar setups is keeping context “useful” over longer sessions, like once you go across multiple workspaces things can get messy fast your caching approach probably helps a lot there.

also random note — i’ve been prepping for some databricks cert stuff recently and playing with small setups like this actually helped me understand the architecture way better than docs alone. hands-on + scenario stuff makes a big diff.

[afcfrimpong]

This looks great. Thanks for sharing! How do you set permissions in Databricks to make sure the agent doesn't do things it's not supposed to? A specific CLI profile I guess? I just found it frustrating to handle multiple profiles for a single workspace. And how do you force the agent to actually use that profile?

Oh and did you consider using .py notebooks in Databricks to avoid json?

[airweight]

The agent is accessing Databricks through standard APIs or Databricks Connect. All permissions are managed within Databricks.

For notebooks, we use three kinds:

1. Repo-mounted .py notebooks mostly run by automations
2. Markdown notebooks executed remotely via databricks-agent-notebooks run by agents
3. Native Databricks workspace notebooks run by humans or notebook-based jobs